:W&Bá¾R_,çM ce5ymtjxª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0®9ø(ƒ‚" 1qm290j78oqj ce5ymtjxŠß ce5ymtjxescheuller-uc-san-diego spatop-sweep"ž _wandb‚{}  batch_size‚1024  dropout‚0.15491252026086438  hidden_dim‚64 ' learning_rate‚0.0008480607180868549  num_embedding_layers‚14Bdpfx_vanilla_ce5ymtjxbnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ ¹—íÒЫ´š3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64ØWÉW; ce5ymtjx*  _wandb‚ {"runtime":0}  _runtime‚0힇bÒÜ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 ¹—íÒðÍÞð*-of*>/data/spatop/options_files/sweep_fixed_dp/config_ce5ymtjx.json*-l* /scratch/logs*-n*dpfx_vanilla_ce5ymtjx2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº 3vsojfx5l0baugi6d4plibppwo2k9ff5¬Î¦ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ÌÛ·ª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º 3vsojfx5l0baugi6d4plibppwo2k9ff5µª½Ç2  requirements.txt·bW ce5ymtjxZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64BSºY ce5ymtjxZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64f«|¿*  ce5ymtjxb trainer/global_step2Ò}`þ\  ce5ymtjxZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64TÁh/  ce5ymtjxb*"trainer/global_step2^õßK  ce5ymtjxj: º—íÒÐÇì±*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] oÛ€“  ce5ymtjxj  º—íÒàîÚåî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ تj\ ce5ymtjxjÊ º—íÒ˜žÐæ·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.371 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 Lb¾q ce5ymtjxj` º—íÒÈÀéæPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). {x;{ ce5ymtjxjj º—íÒðëòæZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ô$ óq ce5ymtjxj` º—íÒ°’ûæPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). d6st ce5ymtjxjc º—íÒø¾ŽçSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). VÁ-n ce5ymtjxj] º—íÒ°¾–çMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). w'Rí ce5ymtjx*Û  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"Ouo”ë ce5ymtjxjÙ º—íÒàà¦êÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. :ë4ÆN ce5ymtjxj¼ º—íÒØ´ê«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. t¡‰ˆë ce5ymtjxjÙ ¼—íÒˆÌæÃÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ºÓóøP ce5ymtjxj¾ ¼—íÒÀ›þí/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. 72}c‚(:† É—íÒŒ‰¤Š gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚181.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¿êG€‚(:ö É—íÒåø‘' proc.memory.availableMB‚ 987.6640625* proc.memory.percent‚15.894770622253418 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚0 network.recv‚42255' disk./.usagePercent‚39.590626902648 disk.drbd2165.in‚0 disk.rbd1.out‚0 cpu‚25.68269647060015$ disk./.usageGB‚589.9289512634277 disk.sda1.out‚0 disk.dm-0.out‚ 3.00146484375 disk.sda3.in‚0 disk.dm-0.in‚0 disk.sda.out‚ 0.3671875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 3.00146484375# proc.memory.rssMB‚ 2604.19921875 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 3.00146484375$ memory_percent‚93.97177696228027 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 0.3671875 disk.sda.in‚0 network.sent‚33842 disk.sda1.in‚0 disk.sda3.out‚ 0.3671875..m3b/ lr-AdamW/pg1"trainer/global_step2:JP€œÝ*3b/ lr-AdamW/pg2"trainer/global_step2:JPªø1ý-) _wandbruntime‚25  _runtime‚25د„쿺 _runtime‚ 25.499369364 _step‚0 & lr-AdamW/pg1‚5.187887039482633e-05 & lr-AdamW/pg2‚5.187887039482633e-05 trainer/global_step‚49 $ _timestamp‚1.784368084055545e+09#¼zÁ¼ & lr-AdamW/pg1‚5.187887039482633e-05 & lr-AdamW/pg2‚5.187887039482633e-05 trainer/global_step‚49 $ _timestamp‚1.784368084055545e+09 _runtime‚ 25.499369364 _step‚0~¹Òc b_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP\Ï=l$bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP VÉói%be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPmæCãg&bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPÍÿÕ5q'bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPò ¹i(be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPü$TOp)bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÖ®Qn*bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP~|ûüm+bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPb,b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPCeç%p-bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP™?†s.bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP¡kw$g/bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPTÜêÀf0bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPŸ#ßÉj1bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP+ g2bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP.;Œ‚n3bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP™ ¦Ðs4bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÃþµön5bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP០t6bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP&áBp7bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPCq`qe8ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP‹2h9bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPb úc:b_ bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP' If?bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPu†ìl@bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP¨Sf’hAbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP­‹™-fBbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPæÚ[dCb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPLDqDbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPÌê*ãkEbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP¢|ÌkiFbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPæ l°sGbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPZ'nHbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPÜbákIbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPëûõ¬_Jb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPe£‘JfKbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPÑ‚­lLbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP*1,aMb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPÈÁÌWgNbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPø¡ŸågObc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPˆ@Âd_Pb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPаLjQbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPqõ.M^RbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPD «€`Sb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP(˜pTbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPï*8ŸpUbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPG³—nrVbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP6Ì#ŠeWba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPü³àenXbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPVªfYbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPWTlnZbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPbmç][bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPV¶Ÿ'j\bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP7p|Új]bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPj:Ùh^bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPçÚÒ¡j_bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP 5wÐi`be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPOyk÷nabj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPEú‹rbbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPß“‚qcbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPÿ=ddb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPµjNnebj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP[Ç‚«sfbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÀÓ&fgbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPï¦ mhbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP ^énibj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP¿÷ljbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPøÓ9vkbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP3Þ^clb_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPŒ}Dßiube BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP† %Ðavb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP• 5µhwbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPÙ(Jòoxbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPbMÒiybe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP-Ò xzbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPP)wk{bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPX‰j|bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPüu45l}bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP„m~bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPð[Z¦`b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPiÔf€ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÛùfmbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP}§Ê²p‚bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPy½a­`ƒb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP襣X„bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP§¤©9p…bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPÝ¢Rw†br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP „Áp‡bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP*_nˆbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPÉj”g‰bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP=Ûµ‹oŠbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP¿_©vm‹bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPRÇ©_ŒbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP+HL‹ibd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÏPò$iŽbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP÷È¥hbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP›Ü¯lbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP-4‡ÿr‘bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPi-+&k’bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP穼ms“bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPèþr”bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPâʈ¡j•be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPJ°æh–bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPZ’Âq—bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPK$;o˜bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPìýŽn™bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPˆ¾Jušbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPr».›) _wandbruntime‚25  _runtime‚25«àôˆ$œðN C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666735 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚827802 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚364997 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11745 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1864272039936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1892950833152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843680840563388e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1861305477668 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚472375 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚225648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 28783561216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 28695832576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11692 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 28695832576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 29975073792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚472375 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚433 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1864255000576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190930432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1832711534592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166026752 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 28593943076 _runtime‚ 25.500372548 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚364997 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300166656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚462805 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚111220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1893055601152 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚336642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚225645 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1790892115456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1761083068416 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1861408652254 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1791083045888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 28783561216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11745 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚223 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚364999 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081460736 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚364999 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1832728573952 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚110997 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 29809047040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11692 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1892950833152 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1861408652254 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1791083045888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 28783561216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11745 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚223 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚364999 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081460736 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚364999 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1832728573952 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚110997 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 29809047040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11692 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1892950833152 GDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1892950833152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843680840563388e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1861305477668 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚472375 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚225648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 28783561216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 28695832576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11692 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 28695832576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 29975073792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚472375 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚433 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1864255000576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190930432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1832711534592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166026752 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 28593943076 _runtime‚ 25.500372548 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚364997 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300166656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0(†‹Rnžbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP ã;@iŸbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP}{Çf ba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPE©]bg¡bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPž”òÍm¢bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPØR¿Ûd£b_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPΔ,Zj¯be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPåK¸øn°bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP¢<æ÷k±bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPÛ1[`²b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPÎ?ÿ r³bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP„Ó:òp´bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP‹„»dµb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP™›7ôeÂb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP¿;£¯oÃbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP—$à¿fÄba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPN/`iÅbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPzY‘¸sÆbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÂîy_ÇbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP/0ƒFlÈbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPУ£âqÉbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPEýÚ#mÊbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPdƒpËbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP]E½˜aÌb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPøÃþÔwÍbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP¼ SmÎbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP§¬lÏbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPdãs—bÐb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPh„oÑbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPx’•†iÒbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPa MØmÓbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP±?„dÔb_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPöiÜbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPž©ºjÝbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP, »Œjîbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÃuï«mïbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP²¦ûoðbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPÝupÂ_ñbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JPë©\Édòb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPNïdl÷bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPçÎE\øbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPˆ ýŒlùbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPÿ Ö1júbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP¨¼kŒgûbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPæÙ ÐVübQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPK‡Wîjýbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPh.ìhþbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPçŽÝeÿb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP)ÕåDh€bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP{ºaqmbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPêÍ”r‚bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP*°"Pcƒb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPkCäj„be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPö´ —k…bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPØ ú+g†bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPFµo'g‡bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPÍÂ)hˆbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP¦PÎ3k‰bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÇÊx9kŠbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPÞ ßDh‹bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPWeèômŒbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP¥¼jffba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPÂëhŽbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP»¢±obj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP8íÕkbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPƒ¹e‘b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPW_>yu’bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPð3¢fr“bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPBl/Çc”b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPzq@|^•bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP3<]–bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP'À¨p—bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPÖÕÊp˜bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPwl?Y.™) _wandbruntime‚25  _runtime‚25£Ùæ:šùL RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 25.842261611 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1930735931904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1930735931904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1901365964800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚113198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 29352927744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163922432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚844304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 $ _timestamp‚1.784368084398242e+09 _step‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1901365964800 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚372218 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚343327 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1898344834526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚372220 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1796019486720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚343557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188826112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1869200609280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1869183569920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1826397374976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚372218 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚853890 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚472086 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 30402791936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1826586201088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 30566714368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚853890 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1901383004160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚481670 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚230129>]²E LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚481670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 29247399944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1930631156736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚433 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1898448009224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚844304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚372220 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÝXæƒ&›ýL M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 25.842261611 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1930735931904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1930735931904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1901365964800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚113198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 29352927744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163922432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚844304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 $ _timestamp‚1.784368084398242e+09 _step‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1901365964800 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚372218 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚343327 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1898344834526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚372220 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1796019486720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚343557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188826112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1869200609280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1869183569920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1826397374976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚372218 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚853890 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚472086 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 30402791936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1826586201088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 30566714368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚853890 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1901383004160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚481670 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚230129 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚481670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 29247399944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1930631156736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚433 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1898448009224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚844304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚372220 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E loss/SRqqt1/assignment_loss"trainer/global_step2:JPǤ»AŸb< loss/FBt1/assignment_loss"trainer/global_step2:JP´',A b< loss/FBt2/assignment_loss"trainer/global_step2:JP?â=@¡b; loss/FRt1/detection_loss"trainer/global_step2:JP[üGËB¢b= loss/SRqqt2/detection_loss"trainer/global_step2:JPAÃ@£b; loss/FBt2/detection_loss"trainer/global_step2:JP¶D=WA¤b< loss/FRt2/assignment_loss"trainer/global_step2:JPóPîýC¥b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP‡ì@¦b; loss/FRt2/detection_loss"trainer/global_step2:JPjuAå7§b2 loss/total_loss"trainer/global_step2:JPª‡B¨b= loss/SRqqt1/detection_loss"trainer/global_step2:JPç¿àÀ@©b; loss/FBt1/detection_loss"trainer/global_step2:JP0÷¡,.ª) _wandbruntime‚25  _runtime‚25 —#ö«ð 0loss/FBt1/assignment_loss‚2.1776029070218406 /loss/FBt2/assignment_loss‚2.131377110610137 /loss/FRt1/detection_loss‚0.7092014032924274 0loss/SRqqt2/detection_loss‚1.954637787607196 /loss/FBt2/detection_loss‚20.371050211424762 0loss/FRt2/assignment_loss‚5.3480188030438285 1loss/SRqqt2/assignment_loss‚4.208796124953728 -loss/FRt2/detection_loss‚1.72119732032407 %loss/total_loss‚74.96701186327601 1loss/SRqqt1/detection_loss‚3.0345085458660668 .loss/FBt1/detection_loss‚24.53128234744072 % _timestamp‚1.7843680843998225e+09 _runtime‚ 25.843357189 _step‚3 /loss/FRt1/assignment_loss‚4.545477915998972 epoch‚0 trainer/global_step‚49 0loss/SRqqt1/assignment_loss‚4.23386138569225˜ÿVÑú¬ô 1loss/SRqqt1/detection_loss‚3.0345085458660668 .loss/FBt1/detection_loss‚24.53128234744072 % _timestamp‚1.7843680843998225e+09 _runtime‚ 25.843357189 _step‚3 /loss/FRt1/assignment_loss‚4.545477915998972 epoch‚0 trainer/global_step‚49 0loss/SRqqt1/assignment_loss‚4.23386138569225 0loss/FBt1/assignment_loss‚2.1776029070218406 /loss/FBt2/assignment_loss‚2.131377110610137 /loss/FRt1/detection_loss‚0.7092014032924274 0loss/SRqqt2/detection_loss‚1.954637787607196 /loss/FBt2/detection_loss‚20.371050211424762 0loss/FRt2/assignment_loss‚5.3480188030438285 1loss/SRqqt2/assignment_loss‚4.208796124953728 -loss/FRt2/detection_loss‚1.72119732032407 %loss/total_loss‚74.96701186327601†×ƒœ­‚(:‘ Ø—íÒ¢ÞþŠ gpu.0.gpu‚59.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 \r»ž®‚(:“ Ø—íÒü˜€ disk.sda3.in‚0 disk.nvme0c0n1.out‚ 8.1484375# proc.memory.rssMB‚ 2604.25390625 disk.rbd0.in‚0 disk.dm-0.in‚ 65.81640625 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 65.81640625* disk./.usagePercent‚39.590665815793194 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚0.0625) proc.memory.percent‚15.89510440826416 cpu‚26.008237402258594 proc.cpu.threads‚17 disk.nvme0n1.out‚ 8.1484375' proc.memory.availableMB‚ 986.4453125 network.recv‚45675 disk.sda2.out‚0 disk.sda3.out‚ 0.59765625 disk.sda.in‚0! disk.nvme0c0n1.in‚ 65.81640625$ disk./.usageGB‚589.9295310974121 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 0.59765625 disk.sda.out‚ 0.59765625 disk.drbd2165.in‚0$ memory_percent‚93.97921562194824 network.sent‚41542 disk.dm-0.out‚ 8.1484375YÖ-‹¯‚(:„ ç—íÒ픋 gpu.0.gpu‚59.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚186.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÎ3§°‚(:œ ç—íÒ³¾û‹ disk.sda2.in‚0 disk.sda.out‚ 0.8046875! disk.nvme0c0n1.in‚ 65.81640625$ memory_percent‚93.97692680358887# disk.nvme0n1.out‚11.04052734375) disk./.usagePercent‚39.59066043963498 disk.rbd0.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 65.81640625 disk.rbd0.in‚0 cpu‚26.165548754024297 network.recv‚53491 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.0625' proc.memory.availableMB‚ 986.8203125) proc.memory.percent‚15.89517593383789 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚11.04052734375 disk.dm-1.out‚ 0.8046875 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ disk./.usageGB‚589.9294509887695 disk.dm-0.in‚ 65.81640625 disk.sda3.out‚ 0.8046875! proc.memory.rssMB‚ 2604.265625 proc.cpu.threads‚17 disk.dm-0.out‚11.04052734375 network.sent‚94874 disk.drbd2165.out‚0dÜp.±) _wandbruntime‚48  _runtime‚48ÄÇ6ò½ ' lr-AdamW/pg1‚0.00010481649324668993 ' lr-AdamW/pg2‚0.00010481649324668993 trainer/global_step‚99 % _timestamp‚1.7843681065775213e+09 _runtime‚ 48.021316999 _step‚4|¹ä}dzÁ _step‚4 ' lr-AdamW/pg1‚0.00010481649324668993 ' lr-AdamW/pg2‚0.00010481649324668993 trainer/global_step‚99 % _timestamp‚1.7843681065775213e+09 _runtime‚ 48.0213169990ç0.´) _wandbruntime‚48  _runtime‚48§W“z'µôN G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 59542350848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3777104037376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3506683703296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3713305223776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3713408398362 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3506658799616 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 57283786240 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚671742 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3776999269376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚928707 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚221673 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 59365838336 _step‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3719803211776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 57196057600 % _timestamp‚1.7843681065785153e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚449848 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚449845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3566024637952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚726049 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚221894 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201416192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3719820251136 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚938277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚928707 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 57196057600 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3566226054144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚726049 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3713305223776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3713408398362 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3506658799616 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 57283786240 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚671742 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3776999269376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚928707 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚221673 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 59365838336 _step‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3719803211776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 57196057600 % _timestamp‚1.7843681065785153e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚449848 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚449845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3566024637952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚726049 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚221894 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201416192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3719820251136 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚938277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚928707 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 57196057600 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3566226054144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚726049 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚678175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3756914176000 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3814784368128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚664 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚947572 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57853152768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 57559244314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚733268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176505344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3756914176000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚733268 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚733270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3601525249536 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3750447755332 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57645379652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57853152768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚678404 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3541570314240 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3750344580634 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚937988 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚947572 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 59954935296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 60131440640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚454332 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 57765416960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201409024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1671256 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚99 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3541595217920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128ßL%¯ˆ&¹‚M R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3814784368128 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚733270 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 57765416960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201409024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1671256 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚99 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3541595217920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚937988 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚224072 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3601726658560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 % _timestamp‚1.7843681069153433e+09 _runtime‚ 48.359379509 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚678175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3756914176000 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3814784368128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚664 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚947572 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57853152768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 57559244314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚733268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176505344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3756914176000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚733268 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚733270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3601525249536 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3750447755332 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57645379652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57853152768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚678404 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3541570314240 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3750344580634 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚937988 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚947572 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 59954935296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 60131440640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚454332 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5661152473600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170221056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5661047705600 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1087099 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 89104114176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843681290962653e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5341168423936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5661047705600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1394609 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5341363548672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1404179 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_startRp—˜Ì/requested_bytes.small_pool.peak‚ 298570279 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1394609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5575368462336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5575368462336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5661152473600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170221056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5661047705600 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1087099 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 89104114176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843681290962653e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5341168423936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5661047705600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1394609 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5341363548672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1404179 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 ÂT¯.Å) _wandbruntime‚70  _runtime‚70b Ñ&ÆŠM VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5376668685312 trainer/global_step‚149 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1094318 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚334444 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 86353377792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174408192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5376867997184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 86043359360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1413474 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚334680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5602447501440 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 86353377792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 86265641984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1413474 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5612479426560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5612479426560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5612462387200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2507794 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚437 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 89522639872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2507794 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1094320 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1094318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1012973 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5287170949120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5602344326742 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1403890 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 86265641984 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _runtime‚ 70.879774653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5516387102720 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2498208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5612462387200 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 85957224022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5698832804352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 % _timestamp‚1.7843681294355712e+09 _step‚10 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1094318 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚334444 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 86353377792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174408192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5376867997184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 86043359360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1413474 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚334680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5602447501440 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 86353377792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 86265641984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1413474 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5612479426560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5612479426560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5612462387200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2507794 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚437 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 89522639872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2507794 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1094320 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1094318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1012973 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large7¸ Ê _pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5287170949120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5602344326742 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1403890 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 86265641984 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _runtime‚ 70.879774653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 Ú†:ó.È) _wandbruntime‚70  _runtime‚70¦Ï…FúÉô /loss/FRt1/assignment_loss‚4.545585231504578 /loss/FRt2/assignment_loss‚5.357451973393823 /loss/FRt1/detection_loss‚0.4095403011716173 /loss/FRt2/detection_loss‚1.3615459028246721 $loss/total_loss‚71.2300278625935 trainer/global_step‚149 % _timestamp‚1.7843681294368143e+09 _runtime‚ 70.880454186 0loss/SRqqt1/detection_loss‚4.954434056496177 1loss/SRqqt2/detection_loss‚1.1124362828517236 -loss/FBt2/detection_loss‚6.53146650573651 _step‚11 1loss/SRqqt1/assignment_loss‚2.756196699979651 2loss/SRqqt2/assignment_loss‚2.5336744100162663 .loss/FBt1/detection_loss‚41.23646478880854 epoch‚0 0loss/FBt1/assignment_loss‚0.2781220568508348 1loss/FBt2/assignment_loss‚0.15310965295911075C5H‘þÊø epoch‚0 0loss/FBt1/assignment_loss‚0.2781220568508348 1loss/FBt2/assignment_loss‚0.15310965295911075 /loss/FRt1/assignment_loss‚4.545585231504578 /loss/FRt2/assignment_loss‚5.357451973393823 /loss/FRt1/detection_loss‚0.4095403011716173 /loss/FRt2/detection_loss‚1.3615459028246721 $loss/total_loss‚71.2300278625935 trainer/global_step‚149 % _timestamp‚1.7843681294368143e+09 _runtime‚ 70.880454186 0loss/SRqqt1/detection_loss‚4.954434056496177 1loss/SRqqt2/detection_loss‚1.1124362828517236 -loss/FBt2/detection_loss‚6.53146650573651 _step‚11 1loss/SRqqt1/assignment_loss‚2.756196699979651 2loss/SRqqt2/assignment_loss‚2.5336744100162663 .loss/FBt1/detection_loss‚41.23646478880854 †1Ž‘Ë‚(:† …˜íÒ·Ä¢‹ gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üóÒå­Ì‚(:¢ …˜íÒ˜êÊŒ disk.dm-1.out‚ 1.40234375 disk.rbd1.in‚0 disk.rbd1.out‚0.125# proc.memory.rssMB‚ 2604.28515625* proc.memory.percent‚15.895295143127441 disk.drbd2165.in‚0# disk.nvme0n1.out‚19.52490234375 network.recv‚63221 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 1.40234375! disk.nvme0c0n1.in‚ 65.81640625* disk./.usagePercent‚39.590666839823335 disk.dm-0.out‚19.52490234375 cpu‚26.3481641761414$ disk./.usageGB‚589.9295463562012 disk.sda2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 65.81640625( proc.memory.availableMB‚ 985.86328125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 1.40234375% disk.nvme0c0n1.out‚19.52490234375$ memory_percent‚93.98276805877686 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 65.81640625 disk.drbd2165.out‚0 network.sent‚147663# ÌQ‘Í‚(:† ”˜íÒõ–Ï‹ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚180.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ ƒ3¨Î‚(: ”˜íÒÃÓœŒ cpu‚26.425050006230524! disk.nvme0n1.out‚ 837.75390625( proc.memory.availableMB‚ 987.42578125 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1.6796875 disk.dm-0.out‚ 837.75390625* proc.memory.percent‚15.895295143127441 proc.cpu.threads‚17 disk.rbd1.out‚0.1875 disk.rbd0.in‚0 disk.dm-1.in‚0 network.recv‚68259) disk./.usagePercent‚39.64397554468573$ disk./.usageGB‚590.7238845825195 disk.sda.in‚0 disk.sda.out‚ 1.6796875 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 122.24609375 disk.sda3.in‚0# proc.memory.rssMB‚ 2604.28515625 network.sent‚194923 disk.sda2.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.out‚ 837.75390625 disk.rbd0.out‚0 disk.dm-0.in‚ 122.24609375 disk.sda3.out‚ 1.6796875# memory_percent‚93.9732313156128" disk.nvme0c0n1.in‚ 122.24609375|äKV.Ï) _wandbruntime‚93  _runtime‚93Ë´I­Äо ' lr-AdamW/pg2‚0.00021069173895041715 trainer/global_step‚199 $ _timestamp‚1.784368151656645e+09 _runtime‚ 93.100520333 _step‚12 ' lr-AdamW/pg1‚0.00021069173895041715©€ÈÑ ' lr-AdamW/pg2‚0.00021069173895041715 trainer/global_step‚199 $ _timestamp‚1.784368151656645e+09 _runtime‚ 93.100520333 _step‚12 ' lr-AdamW/pg1‚0.00021069173895041715 *•<3.Ò) _wandbruntime‚93  _runtime‚93¼BDŽ'ÓˆO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3318230 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1448149 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚437 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚ 93.101739524 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7430899634176 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7417407890578 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1342167 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚443683 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3308658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1870081 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7116472480256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6997835165696 % _timestamp‚1.7843681516577315e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 114284236288 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7545096141824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7430916673536 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 114196507648 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7116273161216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6997810262016 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 118637314560 _step‚13 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1870081 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚898248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚443919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 118462899200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7545096141824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146ÒǾ¹’'ÔŒO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7545200909824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7430899634176 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1860511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199319040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7545096141824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3318230 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1448149 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚437 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚ 93.101739524 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7430899634176 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7417407890578 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1342167 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚443683 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3308658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1870081 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7116472480256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6997835165696 % _timestamp‚1.7843681516577315e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 114284236288 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7545096141824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7430916673536 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 114196507648 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7116273161216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6997810262016 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 118637314560 _step‚13 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1870081 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚898248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚443919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 118462899200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1348595 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7339988869120 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚445866 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 119062805504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7340005908480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1455368 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1879376 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 114853602816 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚902729 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1455370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 119220436480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 114441339068 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7468027637760 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1348821 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 114765867008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7151967116800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚14 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1455370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1879376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7582881240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7032746680320 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3334746 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1869792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚437 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7454344072850 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182534656 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 114355203730 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 114765867008 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7151784582144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚446089 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1455368 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3325160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7032721776640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843681519957888e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7468027637760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7468010598400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚14 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1455370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1879376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7582881240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7032746680320 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3334746 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1869792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚437 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7454344072850 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182534656 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 114355203730 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 114765867008 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7151784582144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚446089 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1455368 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3325160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7032721776640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843681519957888e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7468027637760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7468010598400 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1348595 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7339988869120 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚445866 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 119062805504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7340005908480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1455368 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1879376 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 114853602816 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚902729 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1455370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 119220436480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 114441339068 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7468027637760 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1348821 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 114765867008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192T’‡³.Ø) _wandbruntime‚93  _runtime‚93ÑíiûÙõ -loss/FRt2/detection_loss‚1.71726024423377 1loss/SRqqt2/detection_loss‚1.0587773544085146 %loss/total_loss‚51.03670379082211 epoch‚0 0loss/FRt2/assignment_loss‚4.6156431721375055 2loss/SRqqt2/assignment_loss‚2.1575152292972852 0loss/FRt1/detection_loss‚0.41134651827507585 trainer/global_step‚199 % _timestamp‚1.7843681519977703e+09 _runtime‚ 93.441302933 0loss/SRqqt1/detection_loss‚3.693456234906692 /loss/FBt1/detection_loss‚25.247371351919494 .loss/FBt2/detection_loss‚4.291211081855908 _step‚15 /loss/FRt1/assignment_loss‚4.579916888179041 0loss/SRqqt1/assignment_loss‚2.47793993607066 0loss/FBt1/assignment_loss‚0.5279880045743014 0loss/FBt2/assignment_loss‚0.2582777749638596\A@ÿÚù epoch‚0 0loss/FRt2/assignment_loss‚4.6156431721375055 2loss/SRqqt2/assignment_loss‚2.1575152292972852 0loss/FRt1/detection_loss‚0.41134651827507585 trainer/global_step‚199 % _timestamp‚1.7843681519977703e+09 _runtime‚ 93.441302933 0loss/SRqqt1/detection_loss‚3.693456234906692 /loss/FBt1/detection_loss‚25.247371351919494 .loss/FBt2/detection_loss‚4.291211081855908 _step‚15 /loss/FRt1/assignment_loss‚4.579916888179041 0loss/SRqqt1/assignment_loss‚2.47793993607066 0loss/FBt1/assignment_loss‚0.5279880045743014 0loss/FBt2/assignment_loss‚0.2582777749638596 -loss/FRt2/detection_loss‚1.71726024423377 1loss/SRqqt2/detection_loss‚1.0587773544085146 %loss/total_loss‚51.03670379082211OÛ­š‘Û‚(:† £˜íÒ¿æŠ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚168.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0reٮ܂(:£ £˜íÒúŒ disk.nvme0n1.in‚ 122.24609375 disk.sda1.out‚0! disk.dm-0.out‚842.76611328125 disk.sda3.out‚1.921875 cpu‚26.465251374617313( proc.memory.availableMB‚ 987.01953125 network.sent‚200245$ disk./.usageGB‚590.6972770690918 disk.rbd0.in‚0 disk.sda.out‚1.921875 disk.drbd2165.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚39.6421898921344$ memory_percent‚93.97571086883545 disk.dm-1.in‚0 network.recv‚72292 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚1.921875 disk.sda3.in‚0 disk.rbd1.out‚0.1875& disk.nvme0c0n1.out‚842.76611328125$ disk.nvme0n1.out‚842.76611328125 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2604.29296875" disk.nvme0c0n1.in‚ 122.24609375 disk.rbd0.out‚0 disk.dm-0.in‚ 122.24609375* proc.memory.percent‚15.895342826843262:&`Á0Ý+ _wandbruntime‚115  _runtime‚115U²ÄÞ¾ & lr-AdamW/pg1‚0.0002636293618022807 & lr-AdamW/pg2‚0.0002636293618022807 trainer/global_step‚249 % _timestamp‚1.7843681741565187e+09 _runtime‚ 115.600297079 _step‚16þì¾Èß % _timestamp‚1.7843681741565187e+09 _runtime‚ 115.600297079 _step‚16 & lr-AdamW/pg1‚0.0002636293618022807 & lr-AdamW/pg2‚0.0002636293618022807 trainer/global_step‚249ªÈH0à+ _wandbruntime‚115  _runtime‚115ÎfJ'áŠO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 142271997134 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1809197 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9286447845376 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 142784461312 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1809199 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554642 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9127118600192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1122445 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9286464884736 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2335983 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 142784461312 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1677090 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1122448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9127135639552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 142185861908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 $ _timestamp‚1.784368174157291e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9429144578048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2326413 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2335983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8743410896896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9269304462100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4145182 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8891582338048 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9269407636686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8891389310464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 142696732672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2326413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9429249346048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9429249346048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4135610 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9286447845376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 148003317248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9429144578048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚249<âŽO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554416 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 148171441152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 142696732672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8743410896896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9269304462100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4145182 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8891582338048 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9269407636686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8891389310464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 142696732672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2326413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9429249346048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9429249346048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4135610 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9286447845376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 148003317248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9429144578048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 142271997134 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1809197 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9286447845376 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_poolëJ•¥.allocated‚ 142784461312 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1809199 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554642 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9127118600192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1122445 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9286464884736 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2335983 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 142784461312 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1677090 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1122448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9127135639552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 142185861908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 $ _timestamp‚1.784368174157291e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9429144578048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2326413 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2335983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4152112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1683557 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 148591287808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9323575848960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9306446993656 trainer/global_step‚249 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1683788 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8926888795648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 115.943324611 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197214720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚18 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9323558809600 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚556856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 142753183438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1816420 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8778322411520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4152112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1816418 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9163607674880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1126932 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1816418 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2335694 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚556628 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2345278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172311040 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4161698 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8927086010368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9306343818958 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9323558809600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8778297507840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843681744992979e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 143266092032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1126929 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 143353827840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 143266092032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 148763598848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Úv,Ÿ&å™M MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1816418 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9163607674880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1126932 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1816418 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2335694 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚556628 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2345278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172311040 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4161698 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8927086010368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9306343818958 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9323558809600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8778297507840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843681744992979e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 143266092032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1126929 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 143353827840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 143266092032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 148763598848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4152112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1683557 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 148591287808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9323575848960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9306446993656 trainer/global_step‚249 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1683788 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8926888795648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 115.943324611 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197214720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚18 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9323558809600 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚556856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 142753183438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1816420 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8778322411520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4152112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2—¯ä0æ+ _wandbruntime‚115  _runtime‚115F$¡±ÿçù 0loss/FRt1/assignment_loss‚4.5948951486228164 /loss/FRt2/assignment_loss‚4.570894036063536 /loss/FRt1/detection_loss‚0.4801857910233725 1loss/SRqqt2/detection_loss‚1.4265579859552058 epoch‚0 0loss/FBt2/assignment_loss‚0.3127417408080267 .loss/FRt2/detection_loss‚1.504259042232671 % _timestamp‚1.7843681745009117e+09 _step‚19 2loss/SRqqt1/assignment_loss‚1.6699469745004734 1loss/FBt1/assignment_loss‚0.11523311449796893 0loss/SRqqt1/detection_loss‚2.518928779625854 /loss/FBt1/detection_loss‚24.662619341714308 &loss/total_loss‚51.351809998209816 _runtime‚ 115.944462208 1loss/SRqqt2/assignment_loss‚2.054047078364279 .loss/FBt2/detection_loss‚7.441500964801302 trainer/global_step‚2490Ú6Mèý epoch‚0 0loss/FBt2/assignment_loss‚0.3127417408080267 .loss/FRt2/detection_loss‚1.504259042232671 % _timestamp‚1.7843681745009117e+09 _step‚19 2loss/SRqqt1/assignment_loss‚1.6699469745004734 1loss/FBt1/assignment_loss‚0.11523311449796893 0loss/SRqqt1/detection_loss‚2.518928779625854 /loss/FBt1/detection_loss‚24.662619341714308 &loss/total_loss‚51.351809998209816 _runtime‚ 115.944462208 1loss/SRqqt2/assignment_loss‚2.054047078364279 .loss/FBt2/detection_loss‚7.441500964801302 trainer/global_step‚249 0loss/FRt1/assignment_loss‚4.5948951486228164 /loss/FRt2/assignment_loss‚4.570894036063536 /loss/FRt1/detection_loss‚0.4801857910233725 1loss/SRqqt2/detection_loss‚1.4265579859552058Ì t‹‘é‚(:† ²˜í҇氊 gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A—­ê‚(:¢ ²˜íÒ·µÏŒ network.recv‚77457( disk./.usagePercent‚39.5942711698979 disk.nvme0n1.in‚ 122.24609375 cpu‚26.477372105571888$ disk./.usageGB‚589.9832534790039 disk.sda.in‚0 disk.rbd0.in‚0 network.sent‚247462% disk.nvme0c0n1.out‚849.3505859375 disk.dm-1.in‚0* proc.memory.percent‚15.895342826843262 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 122.24609375 disk.sda1.in‚0 disk.dm-0.in‚ 122.24609375$ memory_percent‚93.97706985473633 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-0.out‚849.3505859375 disk.sda2.in‚0& proc.memory.availableMB‚ 986.796875 proc.cpu.threads‚17 disk.sda.out‚ 2.11328125 disk.drbd2165.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚849.3505859375 disk.rbd0.out‚0 disk.sda3.out‚ 2.11328125 disk.rbd1.out‚0.25 disk.sda2.out‚0 disk.dm-1.out‚ 2.11328125# proc.memory.rssMB‚ 2604.29296875Cßœë‚(:‘ Á˜íÒ…¡ ‹ gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚184.761' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.952200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=à¨ì‚(: Á˜íÒþäóŒ disk.sda3.out‚2.40625 disk.rbd1.out‚0.25$ disk.nvme0n1.out‚851.88232421875 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚851.88232421875 disk.rbd0.in‚0( proc.memory.availableMB‚ 740.43359375) disk./.usagePercent‚39.59248910145206 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚294855" disk.nvme0c0n1.in‚ 122.24609375 disk.rbd0.out‚0$ memory_percent‚95.48075199127197 disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 122.24609375$ disk./.usageGB‚589.9566993713379 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚2.40625 disk.sda.out‚2.40625 disk.nvme0n1.in‚ 122.24609375 disk.sda2.out‚0* proc.memory.percent‚15.895366668701172 network.recv‚82737 disk.sda1.in‚0! disk.dm-0.out‚851.88232421875! proc.memory.rssMB‚ 2604.296875 cpu‚26.508564113730703‰%l0í+ _wandbruntime‚138  _runtime‚138¸vÙÆîÀ ' lr-AdamW/pg1‚0.00031656698465414436 ' lr-AdamW/pg2‚0.00031656698465414436 trainer/global_step‚299 % _timestamp‚1.7843681968895235e+09 _runtime‚ 138.333395942 _step‚20=’ƒÊïÄ ' lr-AdamW/pg1‚0.00031656698465414436 ' lr-AdamW/pg2‚0.00031656698465414436 trainer/global_step‚299 % _timestamp‚1.7843681968895235e+09 _runtime‚ 138.333395942 _step‚20†ûØ0ð+ _wandbruntime‚138  _runtime‚138ßLþoñ›O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11313193014272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1346645 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10666508407296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 177719001088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11121304208208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11313297782272 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11313193014272 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2170249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11121407382794 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2792315 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 VHDeviceStatsMonitor.on_train_batcJ«ŠÁ2h_start/inactive_split_bytes.all.current‚ 197221888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2170247 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1346648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2792315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10488986628096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚299 _runtime‚ 138.334541213 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4962562 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11142013095936S¼}¥'òŸO ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 177546682880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4962562 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11142013095936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11313193014272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1346645 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10666508407296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 177719001088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11121304208208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11313297782272 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11313193014272 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2170249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11121407382794 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2792315 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197221888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2170247 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1346648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2792315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10488986628096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚299 _runtime‚ 138.334541213 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10987209441280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2811180 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚667448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201409024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 171854052864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4988650 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 171237298484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 171151163146 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10702204574720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 178129926656 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11158446739764 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2811180 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11350978113024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11179107020800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11179107020800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10523873239040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2177470 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4979064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 178306432000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 138.676698449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2177468 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176505344 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11158343565066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2018577 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚667678 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4988650 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2177468 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 171854052864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4979064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2018810 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2801596 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11350978113024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11179124060160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4979064 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 178306432000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 138.676698449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2177468 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176505344 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11158343565066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2018577 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚667678 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4988650 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2177468 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 171854052864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4979064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2018810 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚âôwO„0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2801596 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11350978113024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11179124060160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10987209441280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2811180 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚667448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201409024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 171854052864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4988650 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 171237298484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 171151163146 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10702204574720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 178129926656 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11158446739764 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2811180 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11350978113024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11179107020800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11179107020800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10523873239040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0™Æèt0ö+ _wandbruntime‚138  _runtime‚138”a¨­÷ú 1loss/FBt2/assignment_loss‚0.09559987699483624 /loss/FRt2/detection_loss‚1.5101357001966487 /loss/FBt1/detection_loss‚22.737313785654806 %loss/total_loss‚47.34637242309883 trainer/global_step‚299 _runtime‚ 138.677901359 /loss/SRqqt2/detection_loss‚0.96509079595759 /loss/FBt2/detection_loss‚5.7741348570356275 _step‚23 1loss/SRqqt1/detection_loss‚3.8868982928680538 /loss/FRt1/assignment_loss‚4.106342432803886 /loss/FRt2/assignment_loss‚4.086991712119844 2loss/SRqqt1/assignment_loss‚1.9365486455499195 2loss/SRqqt2/assignment_loss‚1.6674582591993892 0loss/FRt1/detection_loss‚0.44864723675532014 epoch‚0 % _timestamp‚1.7843681972343712e+09 0loss/FBt1/assignment_loss‚0.1312108279629077VÙøþ /loss/FRt1/assignment_loss‚4.106342432803886 /loss/FRt2/assignment_loss‚4.086991712119844 2loss/SRqqt1/assignment_loss‚1.9365486455499195 2loss/SRqqt2/assignment_loss‚1.6674582591993892 0loss/FRt1/detection_loss‚0.44864723675532014 epoch‚0 % _timestamp‚1.7843681972343712e+09 0loss/FBt1/assignment_loss‚0.1312108279629077 1loss/FBt2/assignment_loss‚0.09559987699483624 /loss/FRt2/detection_loss‚1.5101357001966487 /loss/FBt1/detection_loss‚22.737313785654806 %loss/total_loss‚47.34637242309883 trainer/global_step‚299 _runtime‚ 138.677901359 /loss/SRqqt2/detection_loss‚0.96509079595759 /loss/FBt2/detection_loss‚5.7741348570356275 _step‚23 1loss/SRqqt1/detection_loss‚3.8868982928680538r(L{‘ù‚(:† ИíÒíéåŠ gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æFIs¨ú‚(: ИíÒ¯äŒ disk.dm-0.out‚857.7919921875 network.recv‚86766 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 219.84765625 disk.sda2.out‚0) disk./.usagePercent‚39.59250702197946 disk.sda3.in‚0 disk.sda3.out‚2.671875 disk.nvme0n1.in‚ 219.84765625# disk.nvme0n1.out‚857.7919921875! proc.memory.rssMB‚ 2604.296875 cpu‚26.52054056387604 proc.cpu.threads‚17 disk.sda.out‚2.671875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚2.671875# memory_percent‚95.4878568649292* proc.memory.percent‚15.895366668701172( proc.memory.availableMB‚ 739.26953125 disk.sda.in‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 network.sent‚300179$ disk./.usageGB‚589.9569664001465 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚0.3125" disk.nvme0c0n1.in‚ 219.84765625% disk.nvme0c0n1.out‚857.7919921875n©Ï0û+ _wandbruntime‚160  _runtime‚160–fÆüÀ % _timestamp‚1.7843682194345567e+09 _runtime‚ 160.878368662 _step‚24 ' lr-AdamW/pg1‚0.00036950460750600795 ' lr-AdamW/pg2‚0.00036950460750600795 trainer/global_step‚349=H­ŸÊýÄ ' lr-AdamW/pg2‚0.00036950460750600795 trainer/global_step‚349 % _timestamp‚1.7843682194345567e+09 _runtime‚ 160.878368662 _step‚24 ' lr-AdamW/pg1‚0.00036950460750600795Þ EØ0þ+ _wandbruntime‚160  _runtime‚160å2ÿ» 'ÿšO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3267787 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203513344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13197241450496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 199784911360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12997544267776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2531297 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚777050 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚25 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2531299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2531297 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 199784911360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12234537455616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1570845 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 207078973952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12973303954316 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3258217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13197346218496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13197241450496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12997561307136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚776825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2347670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12234562359296 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040ÎŽB¤'€žO ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 199784911360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12997544267776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2531297 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚777050 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚25 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2531299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2531297 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 199784911360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12234537455616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1570845 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 207078973952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12973303954316 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3258217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13197346218496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13197241450496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12997561307136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚776825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2347670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12234562359296 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3267787 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203513344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13197241450496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 29857027985¥Ø0+ _wandbruntime‚161  _runtime‚1612*´I‚¥M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2538518 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13010343311174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13234921774080 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1575329 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12810794168320 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3277082 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13034672271360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 % _timestamp‚1.7843682197744071e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5815602 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13235026549248 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2354418 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚26 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2538520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13235026549248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5815602 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2538520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12269448970240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13034655232000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 200266542080 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5806016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12810811207680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 C9DeviceStatsMonitor.on_train_batch_end/allocatÞ‰Ebion.all.peak‚11747 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3267498 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13010446485872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 200354277888 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 199635278192 trainer/global_step‚349 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 200266542080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2538518 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207839575040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚779089 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13034655232000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161825280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5806016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 200354277888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3267498 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12477126720000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12269473873920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186728960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _runtime‚ 161.218524219 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3277082 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186728960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _runtime‚ 161.218524219 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3277082 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13234921774080 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1575329 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12810794168320 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3277082 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13034672271360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 % _timestamp‚1.7843682197744071e+09 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5815602 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13235026549248 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2354418 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚26 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2538520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13235026549248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5815602 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2538520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12269448970240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13034655232000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 200266542080 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5806016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12810811207680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3267498 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13010446485872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 200354277888 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 199635278192 trainer/global_step‚349 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 200266542080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2538518 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207839575040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚779089 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13034655232000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161825280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5806016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 200354277888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3267498 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12477126720000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12269473873920 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1795048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 183.432266099 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 228285136384 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2892349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14853109518336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14216754454016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14597940938752 % _timestamp‚1.7843682419882832e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14853092478976 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚886713 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 228197407744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2681758 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 227379801032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724119 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15081394654720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2892347 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2892347 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14853092478976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3733689 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3733689 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 236807293952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14597923899392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 228285136384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13980113186816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15081289886720 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2681978 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15081394654720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13980138090496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14853109518336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1795045 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15081289886720 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 236641267200 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6616466 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14825406875010 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14825303700424 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2892349 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6626038 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚886930 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146tBm6ŸO ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 236641267200 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6616466 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14825406875010 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14825303700424 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2892349 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6626038 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚886930 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JBDeviceStatsMon˜«”2o!itor.on_train_batch_start/inactive_split.all.current‚220 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1795048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 183.432266099 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 228285136384 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2892349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14853109518336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14216754454016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14597940938752 % _timestamp‚1.7843682419882832e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14853092478976 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚886713 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 228197407744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2681758 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 227379801032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724119 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15081394654720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2892347 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2892347 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14853092478976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3733689 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3733689 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 236807293952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14597923899392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 228285136384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13980113186816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15081289886720 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2681978 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15081394654720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13980138090496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14853109518336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1795045 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15081289886720Aä0‘+ _wandbruntime‚183  _runtime‚1830|_«&’¥M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6632968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2688420 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6632968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6642554 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14862343057282 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚30 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2688649 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15118970210304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚889117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174408192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2899568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 228854502912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15119074985472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14252249263616 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15119074985472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14890220482560 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 228766767104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14015049605120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 237398970368 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 228033257900 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3733400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14890220482560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3742984 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843682423258946e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1799532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146°ø˜¯&“©M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1799532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6632968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2688420 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6632968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6642554 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14862343057282 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚30 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2688649 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15118970210304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚889117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174408192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2899568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 228854502912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15119074985472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14252249263616 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15119074985472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14890220482560 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 228766767104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14015049605120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 237398970368 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 228033257900 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3733400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14890220482560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3742984 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843682423258946e+09©½­ó0”+ _wandbruntime‚183  _runtime‚183ßIj•ý 2loss/SRqqt2/assignment_loss‚1.5462604183562194 1loss/FBt2/assignment_loss‚0.10708596145601144 1loss/SRqqt1/detection_loss‚2.7953054591070146 /loss/FRt1/detection_loss‚0.4104917923448327 .loss/FBt2/detection_loss‚6.940349410941143 &loss/total_loss‚27.925849196895697 _step‚31 2loss/SRqqt1/assignment_loss‚1.7531638277217376 /loss/FRt2/detection_loss‚1.4449021455304714 .loss/FBt1/detection_loss‚4.491508822022092 % _timestamp‚1.7843682423274746e+09 _runtime‚ 183.771019584 0loss/FRt1/assignment_loss‚3.9659349535076753 1loss/FBt1/assignment_loss‚0.18845300386586863 1loss/SRqqt2/detection_loss‚0.9526369666315606 epoch‚0 trainer/global_step‚399 0loss/FRt2/assignment_loss‚3.3297564354110674š9– 2loss/SRqqt1/assignment_loss‚1.7531638277217376 /loss/FRt2/detection_loss‚1.4449021455304714 .loss/FBt1/detection_loss‚4.491508822022092 % _timestamp‚1.7843682423274746e+09 _runtime‚ 183.771019584 0loss/FRt1/assignment_loss‚3.9659349535076753 1loss/FBt1/assignment_loss‚0.18845300386586863 1loss/SRqqt2/detection_loss‚0.9526369666315606 epoch‚0 trainer/global_step‚399 0loss/FRt2/assignment_loss‚3.3297564354110674 2loss/SRqqt2/assignment_loss‚1.5462604183562194 1loss/FBt2/assignment_loss‚0.10708596145601144 1loss/SRqqt1/detection_loss‚2.7953054591070146 /loss/FRt1/detection_loss‚0.4104917923448327 .loss/FBt2/detection_loss‚6.940349410941143 &loss/total_loss‚27.925849196895697 _step‚31RèÏO‘—‚(:† ý˜íÒÿ³–‹ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚174.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì.>«˜‚(:  ý˜íÒƒ™# disk./.usageGB‚589.956787109375 disk.sda.in‚0 disk.nvme0n1.in‚ 219.84765625 disk.dm-1.out‚ 3.44921875 disk.rbd0.in‚0 disk.sda2.out‚0* proc.memory.percent‚15.895390510559082 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚101828 disk.dm-0.in‚ 219.84765625 disk.dm-0.out‚ 869.06640625# disk.nvme0c0n1.out‚ 869.06640625# proc.memory.rssMB‚ 2604.30078125$ memory_percent‚95.49648761749268( proc.memory.availableMB‚ 737.85546875 disk.drbd2165.out‚0 disk.sda3.out‚ 3.44921875 disk.sda.out‚ 3.44921875 disk.sda2.in‚0 network.sent‚400732) disk./.usagePercent‚39.59249498962535 disk.rbd1.out‚0.375" disk.nvme0c0n1.in‚ 219.84765625! disk.nvme0n1.out‚ 869.06640625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 cpu‚26.557826410335252•ŽA0™+ _wandbruntime‚205  _runtime‚205[ÓÚ½ _runtime‚ 205.901686662 _step‚32 & lr-AdamW/pg1‚0.0004753798532097352 & lr-AdamW/pg2‚0.0004753798532097352 trainer/global_step‚449 $ _timestamp‚1.784368264454501e+09‘é Ç›Á & lr-AdamW/pg1‚0.0004753798532097352 & lr-AdamW/pg2‚0.0004753798532097352 trainer/global_step‚449 $ _timestamp‚1.784368264454501e+09 _runtime‚ 205.901686662 _step‚32 %$bV0œ+ _wandbruntime‚205  _runtime‚205‰ô®E›O TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3253397 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16708657729536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚998371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3017619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.la¸ûmR\$rge_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2019245 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16965338322944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16708657729536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 % _timestamp‚1.7843682644586914e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7443418 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3253399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 255863915966 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2019248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16708640690176 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4190021 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 256697632768 trainer/global_step‚449 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _step‚33 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 256785361408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16677406621118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16965443090944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15991860272640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3253399 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16421525665792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4199591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694¬­({¥'žŸO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3253399 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16421525665792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4199591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3253397 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16708657729536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚998371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3017619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2019245 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16965338322944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16708657729536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 % _timestamp‚1.7843682644586914e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7443418 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3253399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 255863915966 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2019248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16708640690176 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4190021 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 256697632768 trainer/global_step‚449 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _step‚33 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 256785361408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16677406621118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16965443090944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15991860272640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944!™î¶×0Ÿ+ _wandbruntime‚206  _runtime‚206Ä7w­& §M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2023729 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7469506 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16027554297856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16745768693760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7459920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2023732 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17003018646528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 257266992128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16745751654400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16457997701120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190923264 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16745751654400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 256345102270 _step‚34 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7469506 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3260618 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4208886 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3024277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚449 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1000320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 257266992128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3260620 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3024049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15760600432640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16714445978088 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7459920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 266928961536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1000545 _runtime‚ 206.234871729 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16745768693760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 % _timestamp‚1.7843682647908342e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4208886 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4199302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 256431237608QÙ…èÿ ¡«M H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 256345102270 _step‚34 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7469506 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3260618 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4208886 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3024277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚449 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1000320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 257266992128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3260620 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3024049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15760600432640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16714445978088 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7¾5x!²459920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 266928961536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1000545 _runtime‚ 206.234871729 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16745768693760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 % _timestamp‚1.7843682647908342e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4208886 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4199302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 256431237608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2023729 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7469506 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16027554297856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16745768693760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7459920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2023732 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17003018646528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 257266992128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16745751654400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16457997701120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190923264 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16745751654400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338"¼b^0¢+ _wandbruntime‚206  _runtime‚206t¡ô0ý£÷ .loss/FBt2/detection_loss‚2.081031419761483 _step‚35 1loss/SRqqt1/detection_loss‚3.2633427770578205 epoch‚0 2loss/SRqqt2/assignment_loss‚1.6257821921064954 /loss/FRt1/detection_loss‚0.4866437562816579 -loss/FBt1/detection_loss‚8.55211540352154 _runtime‚ 206.235946618 0loss/FRt1/assignment_loss‚3.9704582047733394 /loss/FRt2/assignment_loss‚3.080449273966361 /loss/FBt1/assignment_loss‚0.136628454545202 %loss/total_loss‚27.49083280226224 trainer/global_step‚449 % _timestamp‚1.7843682647924106e+09 2loss/SRqqt1/assignment_loss‚1.7802927321195603 0loss/FBt2/assignment_loss‚0.2498803479325592 /loss/FRt2/detection_loss‚1.4332223750407929 1loss/SRqqt2/detection_loss‚0.8309858651554225§@§¤û 2loss/SRqqt2/assignment_loss‚1.6257821921064954 /loss/FRt1/detection_loss‚0.4866437562816579 -loss/FBt1/detection_loss‚8.55211540352154 _runtime‚ 206.235946618 0loss/FRt1/assignment_loss‚3.9704582047733394 /loss/FRt2/assignment_loss‚3.080449273966361 /loss/FBt1/assignment_loss‚0.136628454545202 %loss/total_loss‚27.49083280226224 trainer/global_step‚449 % _timestamp‚1.7843682647924106e+09 2loss/SRqqt1/assignment_loss‚1.7802927321195603 0loss/FBt2/assignment_loss‚0.2498803479325592 /loss/FRt2/detection_loss‚1.4332223750407929 1loss/SRqqt2/detection_loss‚0.8309858651554225 .loss/FBt2/detection_loss‚2.081031419761483 _step‚35 1loss/SRqqt1/detection_loss‚3.2633427770578205 epoch‚0#vêÙš¥‚(:„ Œ™íÒŒ™‡‹ gpu.0.gpu‚66.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨]¬±¦‚(:¦ Œ™íÒ½à…$ disk./.usageGB‚589.9574012756348 disk.drbd2165.in‚0( proc.memory.availableMB‚ 738.30859375$ disk.nvme0n1.out‚880.41845703125# memory_percent‚95.4937219619751 disk.sda3.out‚ 4.5859375 disk.sda.in‚0& disk.nvme0c0n1.out‚880.41845703125 disk.nvme0n1.in‚ 302.91796875 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0* proc.memory.percent‚15.895390510559082! disk.dm-0.out‚880.41845703125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 302.91796875 cpu‚26.57524921118732 disk.sda1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2604.30078125 network.sent‚448446 disk.dm-1.out‚ 4.5859375 disk.dm-0.in‚ 302.91796875 disk.sda.out‚ 4.5859375 network.recv‚107043) disk./.usagePercent‚39.59253620683836 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.4375 proc.cpu.threads‚17 disk.rbd0.out‚0Zg{§‚(:… ›™íÒ®ŒÇ‹ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚193.885' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.777 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹y¤ª¨‚(:Ÿ ›™íÒÍ•¨Œ disk.rbd0.out‚0* proc.memory.percent‚15.895390510559082 disk.sda2.out‚0# proc.memory.rssMB‚ 2604.30078125 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚6.140625) disk./.usagePercent‚39.59253595083083 disk.sda1.out‚0$ disk./.usageGB‚589.9573974609375 disk.dm-0.out‚ 883.677734375 disk.sda3.in‚0 disk.sda.out‚6.140625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 302.91796875" disk.nvme0n1.out‚ 883.677734375 network.sent‚496094 disk.dm-0.in‚ 302.91796875 disk.drbd2165.in‚0 disk.rbd1.out‚0.4375 disk.nvme0n1.in‚ 302.91796875( proc.memory.availableMB‚ 738.01953125 cpu‚26.594689367891213 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 883.677734375 disk.sda2.in‚0$ memory_percent‚95.49548625946045 network.recv‚112992 disk.sda3.out‚6.140625&‚“0©+ _wandbruntime‚228  _runtime‚228¯5ððĪ¾ & lr-AdamW/pg1‚0.0005283174760615987 & lr-AdamW/pg2‚0.0005283174760615987 trainer/global_step‚499 % _timestamp‚1.7843682869215274e+09 _runtime‚ 228.365321159 _step‚365|xÈ«Â _step‚36 & lr-AdamW/pg1‚0.0005283174760615987 & lr-AdamW/pg2‚0.0005283174760615987 trainer/global_step‚499 % _timestamp‚1.7843682869215274e+09 _runtime‚ 228.365321159$–'n„0¬+ _wandbruntime‚228  _runtime‚228tI^f£'­O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18245127432192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4665493 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 285285586432 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3614449 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3614447 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17767195023360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3352327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18529406367226 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚ 228.366581077 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17471289552896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18245144471552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 284261895674 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8279942 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18529303192640 trainer/global_step‚499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 295905470464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4655923 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3352550 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18849386759168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4665493 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8270370 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166026752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2243448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17767004092928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18564188901376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2243445 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 284175760448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192>P8µ§'®¡O K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4655923 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3352550 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18849386759168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4665493 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8270370 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166026752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2243448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17767004092928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18564188901376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2243445 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 284175760448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18245127432192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4665493 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 285285586432 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3614449 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3614447 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17767195023360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3352327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18529406367226 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚ 228.366581077 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17471289552896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18245144471552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 284261895674 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8279942 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18529303192640 trainer/global_step‚499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 295905470464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0%9:å?0¯+ _wandbruntime‚228  _runtime‚228óf’ˆÐ°§M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18281599467520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8286872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17802500837888 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18566445724196 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 284829217316 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8296458 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 285767217152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.cuõ †yÝ"rrent‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3621670 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 285854952960 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8286872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17506176163840 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18887171857920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 285767217152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1111284 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4665204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 296324674048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1111046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3621668 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18601299865600 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4665204 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2247932 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 % _timestamp‚1.7843682872621734e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8296458 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3621670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18601316904960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18887067082752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3358975 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18281616506880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146ü+w±&±«M WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 285767217152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1111284 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4665204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 296324674048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1111046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3621668 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18601299865600 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4665204 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2247932 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 % _timestamp‚1.7843682872621734e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8296458 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3621670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18601316904960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18887067082752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3358975 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18281616506880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18281599467520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8286872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17802500837888 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18566445724196 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 284829217316 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8296458 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 285767217152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3621670 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 285854952960 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8286872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17506176163840 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18887171857920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338&6äô0²+ _wandbruntime‚228  _runtime‚228³ã³ú 2loss/SRqqt2/assignment_loss‚1.4857853752558519 /loss/FRt1/detection_loss‚0.4222241502396157 epoch‚0 _runtime‚ 228.707255295 1loss/FBt1/assignment_loss‚0.11708530508741578 /loss/FRt2/detection_loss‚1.3728401050971182 1loss/SRqqt2/detection_loss‚0.8271741705924478 % _timestamp‚1.7843682872637103e+09 0loss/FRt1/assignment_loss‚4.0119770559982655 1loss/FBt2/assignment_loss‚0.14655550976446344 .loss/FBt1/detection_loss‚5.000287070503243 .loss/FBt2/detection_loss‚3.517161854879248 trainer/global_step‚499 _step‚39 0loss/FRt2/assignment_loss‚2.9163742803059014 2loss/SRqqt1/assignment_loss‚1.8990873596817255 0loss/SRqqt1/detection_loss‚2.591724816071801 $loss/total_loss‚24.3082770534771š›´þ 1loss/FBt1/assignment_loss‚0.11708530508741578 /loss/FRt2/detection_loss‚1.3728401050971182 1loss/SRqqt2/detection_loss‚0.8271741705924478 % _timestamp‚1.7843682872637103e+09 0loss/FRt1/assignment_loss‚4.0119770559982655 1loss/FBt2/assignment_loss‚0.14655550976446344 .loss/FBt1/detection_loss‚5.000287070503243 .loss/FBt2/detection_loss‚3.517161854879248 trainer/global_step‚499 _step‚39 0loss/FRt2/assignment_loss‚2.9163742803059014 2loss/SRqqt1/assignment_loss‚1.8990873596817255 0loss/SRqqt1/detection_loss‚2.591724816071801 $loss/total_loss‚24.3082770534771 2loss/SRqqt2/assignment_loss‚1.4857853752558519 /loss/FRt1/detection_loss‚0.4222241502396157 epoch‚0 _runtime‚ 228.707255295'%ùVŽµ‚(:ƒ ª™íÒþÂአgpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚176.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.23 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÝíÞ±¶‚(:¦ ª™íÒ( proc.memory.availableMB‚ 737.93359375$ disk./.usageGB‚590.7279853820801 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 302.91796875 disk.dm-1.in‚0$ memory_percent‚95.49601078033447 cpu‚26.602301117399563 network.sent‚501396% disk.nvme0c0n1.out‚919.6416015625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 302.91796875 disk.drbd2165.out‚0 disk.sda2.in‚0 network.recv‚116802* disk./.usagePercent‚39.644250752785034 disk.sda3.out‚ 6.51953125 disk.rbd1.out‚0.5 disk.sda2.out‚0 disk.dm-1.out‚ 6.51953125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.out‚ 6.51953125" disk.nvme0c0n1.in‚ 302.91796875 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2604.30078125 disk.rbd1.in‚0 disk.dm-0.out‚919.6416015625# disk.nvme0n1.out‚919.6416015625* proc.memory.percent‚15.895390510559082ea•J0·+ _wandbruntime‚250  _runtime‚250îs­Ã¸½ _step‚40 & lr-AdamW/pg1‚0.0005812550989134624 & lr-AdamW/pg2‚0.0005812550989134624 trainer/global_step‚549 $ _timestamp‚1.784368309442486e+09 _runtime‚ 250.886238815ν¯ŒÇ¹Á _runtime‚ 250.886238815 _step‚40 & lr-AdamW/pg1‚0.0005812550989134624 & lr-AdamW/pg2‚0.0005812550989134624 trainer/global_step‚549 $ _timestamp‚1.784368309442486e+09(S 0º+ _wandbruntime‚250  _runtime‚250\ƒIö£'»O L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3687652 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 168123904 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20733539963392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 313785811456 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5121825 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19216840380416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20419737112576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1220007 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 312573740156 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚ 250.887456666 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9106894 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5131395 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2467648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20733539963392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 325268853760 _step‚41 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20733435195392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 313698082816 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20419754151936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20733435195392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20419737112576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2467645 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20068746237952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3975497 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19542109234176 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5121825 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 313698082816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20068729198592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3975497 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20381302938748 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3975499 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19542302261760 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3687888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19216865284096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 325436977664 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 312659875382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 B¼¡O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.l‹Md&arge_pool.allocated‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5131395 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚549 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20381406113334 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9097322 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 313785811456 % _timestamp‚1.7843683094434612e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20419754151936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20733435195392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20419737112576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2467645 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20068746237952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3975497 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19542109234176 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5121825 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 313698082816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20068729198592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3975497 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20381302938748 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3975499 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19542302261760 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3687888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19216865284096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 325436977664 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 312659875382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚ 250.887456666 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9106894 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5131395 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2467648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20733539963392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 325268853760 _step‚41 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20733435195392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 313698082816 H8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2472129 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19251751895040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1222520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20105201233920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3982720 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 314355177984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190923264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 314267442176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚549 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 314267442176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19577804221952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19577613298688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5140690 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3982720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20105218273280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9123410 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20771115518976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20456848076800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3694652 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19251776798720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3694414 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5131106 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9123410 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 313141061686 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20771220294144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 325861403648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 313227197024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚ 251.221301305 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 $ _timestamp‚1.784368309777278e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9113824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166019584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3982718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 314355177984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0˜Æ²"°&¿ªM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 314267442176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19577804221952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19577613298688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5140690 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3982720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20105218273280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9123410 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20771115518976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20456848076800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3694652 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19251776798720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3694414 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5131106 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9123410 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 313141061686 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20771220294144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 325861403648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 313227197024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚ 251.221301305 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 $ _timestamp‚1.784368309777278e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9113824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166019584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3982718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 314355177984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9113824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2472129 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19251751895040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1222520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20105201233920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3982720 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 314355177984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190923264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 314267442176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚549*º¸–È0À+ _wandbruntime‚251  _runtime‚251w"þfÁü 2loss/SRqqt1/assignment_loss‚1.3858707012651965 0loss/FRt1/detection_loss‚0.46853026164034195 /loss/FBt2/detection_loss‚2.4958961774434756 &loss/total_loss‚22.604976516051426 epoch‚0 _step‚43 0loss/FRt1/assignment_loss‚3.9294932327460295 2loss/SRqqt2/assignment_loss‚1.5575547005856436 .loss/FBt1/detection_loss‚4.025968110049144 % _timestamp‚1.7843683097789297e+09 0loss/FRt2/assignment_loss‚2.6661612032449735 0loss/FBt1/assignment_loss‚0.2467247561683962 /loss/FRt2/detection_loss‚1.3597455680746808 trainer/global_step‚549 1loss/FBt2/assignment_loss‚0.24329379688242625 0loss/SRqqt1/detection_loss‚3.503440959703957 1loss/SRqqt2/detection_loss‚0.7222970482471631 _runtime‚ 251.22274371²Òø€ 0loss/FRt2/assignment_loss‚2.6661612032449735 0loss/FBt1/assignment_loss‚0.2467247561683962 /loss/FRt2/detection_loss‚1.3597455680746808 trainer/global_step‚549 1loss/FBt2/assignment_loss‚0.24329379688242625 0loss/SRqqt1/detection_loss‚3.503440959703957 1loss/SRqqt2/detection_loss‚0.7222970482471631 _runtime‚ 251.22274371 2loss/SRqqt1/assignment_loss‚1.3858707012651965 0loss/FRt1/detection_loss‚0.46853026164034195 /loss/FBt2/detection_loss‚2.4958961774434756 &loss/total_loss‚22.604976516051426 epoch‚0 _step‚43 0loss/FRt1/assignment_loss‚3.9294932327460295 2loss/SRqqt2/assignment_loss‚1.5575547005856436 .loss/FBt1/detection_loss‚4.025968110049144 % _timestamp‚1.7843683097789297e+09+º`Cò‘Â(:† ¹™íÒĤ¯‹ gpu.0.gpu‚62.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+>²Ä‚(:§ ¹™íÒÁê÷Œ network.sent‚549033 disk.dm-0.in‚ 302.91796875 disk.drbd2165.in‚0( proc.memory.availableMB‚ 737.75390625 proc.cpu.threads‚17 network.recv‚122690$ disk./.usageGB‚590.9094276428223 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚1790.4150390625$ disk.nvme0n1.out‚1790.4150390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 302.91796875 disk.dm-1.out‚ 6.99609375 disk.rbd1.in‚0$ memory_percent‚95.49710750579834* proc.memory.percent‚15.895390510559082 cpu‚26.61901002691128! disk.dm-0.out‚1790.4150390625 disk.sda3.out‚ 6.99609375 disk.sda.out‚ 6.99609375 disk.rbd1.out‚0.5) disk./.usagePercent‚39.65642749514364 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 302.91796875 disk.sda1.in‚0# proc.memory.rssMB‚ 2604.30078125+q܆#Å‚(:© È™íÒïÐÖŒ networ*"ï‘k.sent‚596467 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 10.35546875 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚15.895438194274902% proc.memory.availableMB‚ 737.65625) disk./.usagePercent‚39.65048300019858 disk.drbd2165.out‚0 disk.rbd1.out‚0.625 disk.dm-0.in‚ 403.22265625$ disk.nvme0n1.out‚1828.0283203125 disk.dm-1.in‚0! disk.dm-0.out‚1828.0283203125 disk.rbd0.in‚0# memory_percent‚95.4977035522461 network.recv‚127853 disk.nvme0n1.in‚ 403.22265625 disk.sda1.out‚0$ disk./.usageGB‚590.8208503723145 disk.sda3.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚1828.0283203125 disk.sda1.in‚0# proc.memory.rssMB‚ 2604.30859375 cpu‚26.627338778864825 disk.sda3.out‚ 10.35546875" disk.nvme0c0n1.in‚ 403.22265625 disk.sda2.in‚0 disk.dm-1.out‚ 10.35546875–ßŜƂ(:‘ È™íҾ‘Œ gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÊo0Ç+ _wandbruntime‚273  _runtime‚273" „9Âȼ _step‚44 % lr-AdamW/pg1‚0.000634192721765326 % lr-AdamW/pg2‚0.000634192721765326 trainer/global_step‚599 % _timestamp‚1.7843683320896122e+09 _runtime‚ 273.533523069¬n¤`ÆÉÀ _runtime‚ 273.533523069 _step‚44 % lr-AdamW/pg1‚0.000634192721765326 % lr-AdamW/pg2‚0.000634192721765326 trainer/global_step‚599 % _timestamp‚1.7843683320896122e+09, Š …0Ê+ _wandbruntime‚273  _runtime‚273.:é¿¢'ËœO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22617483631616 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4336549 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5597297 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5587727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22275302363136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22617588399616 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4336547 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 355003674112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20962416111616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9933846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20962441015296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 273.534671455 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22275285323776 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4022388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4336547 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22617483631616 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21317444689408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1330540 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21892330964992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2691845 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5587727 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 342198307840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 trainer/global_step‚599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 $ _timestamp‚1.784368332090605e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22233302684856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21892348004352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195124736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 354833453056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 340971719864 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 342286036480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170221056 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22233405859442 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4336549 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22617588399616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 342198307840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1330313 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22275302363136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚45 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9924274 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5597297 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 341057855090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4022158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21317249564672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 342286036480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2691848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21892348004352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195124736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 354833453056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 340971719864 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 342286036480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170221056 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22233405859442 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4336549 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22617588399616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 342198307840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1330313 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22275302363136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚45 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9924274 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5597297 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 341057855090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4022158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21317249564672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 342286036480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2691848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22617483631616 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4336549 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5597297 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5587727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22275302363136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22617588399616 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4336547 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 355003674112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20962416111616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9933846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20962441015296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 273.534671455 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22275285323776 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4022388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4336547 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22617483631616 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21317444689408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1330540 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21892330964992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2691845 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5587727 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 342198307840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 trainer/global_step‚599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360-s¼@0Í+ _wandbruntime‚273  _runtime‚273ÔŸ»j­&ΧM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚671 _runtime‚ 273.878062079 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843683324340236e+09 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20997352529920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2696329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1332505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172311040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4343770 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22655163955200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4343768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22312413327360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22312396288000 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22270342041714 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21928803000320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 341539041394 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 342767667200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21352946977280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197214720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20997327626240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9940776 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9950362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2696332 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4343770 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21352749762560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1332745 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5597008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5606592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22312396288000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 342855403008 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5606592 trainer/global_step‚599 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 342855403008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4028834 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 355422136320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21928820039680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9950362 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5597008 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22655268730368 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9940776 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9950362 D>DeviceStatsMonitor.onR'Î8$_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2696332 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4343770 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21352749762560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1332745 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5597008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5606592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22312396288000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 342855403008 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5606592 trainer/global_step‚599 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 342855403008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4028834 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 355422136320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21928820039680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9950362 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5597008 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22655268730368 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20997352529920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2696329 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1332505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172311040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4343770 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22655163955200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4343768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22312413327360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22312396288000 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22270342041714 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21928803000320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 341539041394 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 342767667200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21352946977280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197214720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20997327626240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23715932731392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24130833534976 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 370786261504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24130833534976 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23092570640896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22708016746496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 369455834798 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6053629 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4357236 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 384383673344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4697599 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22707991842816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369369699572 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚ 296.160372033 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24130850574336 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370698532864 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 384553894400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24501532067840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4697597 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195124736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4697597 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10751226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6063199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170221056 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 370698532864 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6063199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4357461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 " _timestamp‚1.7843683547164e+09²q¥q&Ú¢O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10760798 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24501636835840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 369455834798 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6053629 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4357236 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 384383673344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4697599 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22707991842816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183442944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369369699572 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚ 296.160372033 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24130850574336 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370698532864 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 384553894400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24501532067840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4697597 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195124736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4697597 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10751226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6063199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170221056 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 370698532864 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6063199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4357461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 " _timestamp‚1.7843683547164e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23715932731392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24130833534976 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 370786261504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24130833534976 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23092570640896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22708016746496 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24539212391424 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4704818 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4704818 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6062910 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚50 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23752421806080 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10767728 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6072494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6062910 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1443638 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10777314 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24167961538560 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 371355628032 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4363935 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 371355628032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22742928261120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6072494 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 370023156440 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 296.506754638 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23128067685888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24167961538560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 369937021102 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23752404766720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10777314 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2920529 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1443406 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24122444962520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4704820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24167944499200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22742903357440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936ïìÔ„´&Ý®M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10767728 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6072494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6062910 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1443638 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10777314 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24167961538560 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 371355628032 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4363935 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 371355628032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22742928261120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6072494 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11694 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 370023156440 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 296.506754638 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23128067685888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183442944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24167961538560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 369937021102 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23752404766720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10777314 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2920529 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1443406 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24122444962520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4704820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24167944499200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22742903357440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24539317166592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24539212391424 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4704818 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4704818 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6062910 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚50 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23752421806080 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 68451041282'0Þ+ _wandbruntime‚296  _runtime‚296 Ébßú 0loss/FRt1/assignment_loss‚3.9219401988602636 1loss/SRqqt1/assignment_loss‚1.739016758455407 1loss/FBt2/assignment_loss‚0.18603166775908903 /loss/FRt1/detection_loss‚0.4855966056929901 _runtime‚ 296.507869571 1loss/SRqqt2/detection_loss‚0.6278064358900027 0loss/SRqqt1/detection_loss‚2.377648440145311 .loss/FBt1/detection_loss‚4.136519949752837 /loss/FBt2/detection_loss‚0.6781643124146425 %loss/total_loss‚19.50840594912639 epoch‚0 % _timestamp‚1.7843683550643187e+09 _step‚51 0loss/FRt2/assignment_loss‚2.4895556026610777 2loss/SRqqt2/assignment_loss‚1.3695196700446746 0loss/FBt1/assignment_loss‚0.1791359660259468 /loss/FRt2/detection_loss‚1.3174703414241473 trainer/global_step‚649a_ºàþ 0loss/SRqqt1/detection_loss‚2.377648440145311 .loss/FBt1/detection_loss‚4.136519949752837 /loss/FBt2/detection_loss‚0.6781643124146425 %loss/total_loss‚19.50840594912639 epoch‚0 % _timestamp‚1.7843683550643187e+09 _step‚51 0loss/FRt2/assignment_loss‚2.4895556026610777 2loss/SRqqt2/assignment_loss‚1.3695196700446746 0loss/FBt1/assignment_loss‚0.1791359660259468 /loss/FRt2/detection_loss‚1.3174703414241473 trainer/global_step‚649 0loss/FRt1/assignment_loss‚3.9219401988602636 1loss/SRqqt1/assignment_loss‚1.739016758455407 1loss/FBt2/assignment_loss‚0.18603166775908903 /loss/FRt1/detection_loss‚0.4855966056929901 _runtime‚ 296.507869571 1loss/SRqqt2/detection_loss‚0.62780643589000273z,šá‚(: æ™íÒöø“Š gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚137.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.560000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pƒz©â‚(:ž æ™íÒ‰•³Œ disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 11.45703125& proc.memory.availableMB‚ 737.640625# proc.memory.rssMB‚ 2604.30859375 network.sent‚649085 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚0.6875 cpu‚26.64090440662155# disk./.usageGB‚590.005672454834 disk.sda3.out‚ 11.45703125 disk.nvme0n1.in‚ 403.22265625 proc.cpu.threads‚17 disk.dm-0.in‚ 403.22265625) disk./.usagePercent‚39.59577572617663 disk.rbd0.out‚0 disk.dm-1.out‚ 11.45703125" disk.nvme0c0n1.in‚ 403.22265625* proc.memory.percent‚15.895438194274902 network.recv‚136865 disk.dm-1.in‚0 disk.dm-0.out‚ 1901.328125" disk.nvme0c0n1.out‚ 1901.328125$ memory_percent‚95.49779891967773 disk.nvme0n1.out‚ 1901.328125 &[‘ã‚(:† õ™íÒ•åæ‹ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚133.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÂt ªä‚(:Ÿ õ™íÒÈ¥ˆ# disk.nvme0c0n1.out‚ 1904.5703125! disk.nvme0n1.out‚ 1904.5703125 disk.rbd0.out‚0 disk.dm-0.in‚ 403.22265625 network.sent‚696443" disk.nvme0c0n1.in‚ 403.22265625 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda.out‚ 11.75390625# memory_percent‚95.4993486404419* proc.memory.percent‚15.895462036132812 cpu‚26.64265370706682 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 11.75390625 proc.cpu.threads‚17) disk./.usagePercent‚39.59578263838006 disk.sda1.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2604.3125$ disk./.usageGB‚590.0057754516602 disk.nvme0n1.in‚ 403.22265625 disk.dm-0.out‚ 1904.5703125( proc.memory.availableMB‚ 737.38671875 disk.rbd1.in‚0 disk.sda2.out‚0 network.recv‚142207 disk.drbd2165.in‚0 disk.dm-1.out‚ 11.75390625 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.out‚0.6875­(7Ž0å+ _wandbruntime‚318  _runtime‚318Ö¹cÄÄæ¾ trainer/global_step‚699 % _timestamp‚1.7843683773995562e+09 _runtime‚ 318.848877204 _step‚52 & lr-AdamW/pg1‚0.0007400679674690532 & lr-AdamW/pg2‚0.0007400679674690532ÌÆÈç & lr-AdamW/pg1‚0.0007400679674690532 & lr-AdamW/pg2‚0.0007400679674690532 trainer/global_step‚699 % _timestamp‚1.7843683773995562e+09 _runtime‚ 318.848877204 _step‚524·_ñd0è+ _wandbruntime‚318  _runtime‚318WH=ú'é¡O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5058647 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6519531 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5058647 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6529101 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5058649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185539072 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 399286486528 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 397767679280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26385685272064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4692669 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26385580504064 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25986381746176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11587750 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3140245 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25539551537152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26385580504064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1552421 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3140248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚k¾€  306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚210 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25986398785536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚699 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11578178 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 399286486528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24867670083072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24453567574016 _step‚53 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚207 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25937405351658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24453592477696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25539534497792 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473!eáæ«'ê¥O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5058649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25986398785536 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26385685272064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 399198757888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26385580504064 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25986381746176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11587750 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3140245 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25539551537152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26385580504064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1552421 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚671 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3140248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193027584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚210 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25986398785536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚699 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11578178 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 399286486528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚444 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24867670083072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24453567574016 _step‚53 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚207 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25937405351658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24453592477696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25539534497792 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5058647 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6519531 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5058647 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6529101 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5058649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185539072 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 399286486528 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 397767679280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26385685272064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4692669 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4699132 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 414669947392 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 414489247744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5065870 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3144732 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26423365602816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6528812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _runtime‚ 319.19367247 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11604266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25974341533930 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 398335000810 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11604266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180699648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26423365602816 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 399768117248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185539072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26023509749760 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚54 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3144729 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205603328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5065870 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4699365 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25576023572480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6538396 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6538396 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1554633 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26423260827648 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5065868 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1554403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473Èý3€´&í®M EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11604266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25974341533930 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 398335000810 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11604266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180699648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26423365602816 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 399768117248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185539072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26023509749760 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚54 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3144729 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205603328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5065870 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4699365 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25576023572480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6538396 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6538396 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1554633 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26423260827648 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5065868 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1554403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5065868 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6528812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25974444708628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25576006533120 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 398421136148 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4699132 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 414669947392 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 414489247744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5065870 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3144732 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26423365602816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6528812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _runtime‚ 319.193672476fÒsh0î+ _wandbruntime‚319  _runtime‚319ý­æ]àïý /loss/FRt1/detection_loss‚0.6175986351959097 /loss/FBt2/detection_loss‚0.9443253654186333 trainer/global_step‚699 % _timestamp‚1.7843683777516398e+09 1loss/FBt2/assignment_loss‚0.23073485293228996 ïþöp#1loss/SRqqt2/detection_loss‚0.6560458942140448 &loss/total_loss‚17.959855183399974 epoch‚0 0loss/FRt2/assignment_loss‚2.4110897795819657 1loss/FBt1/assignment_loss‚0.26586317923479796 /loss/FRt2/detection_loss‚1.3179435724131308 _runtime‚ 319.195177202 _step‚55 0loss/FRt1/assignment_loss‚3.9567158648173013 2loss/SRqqt1/assignment_loss‚1.6083147567089484 1loss/SRqqt2/assignment_loss‚1.340362674803259 1loss/SRqqt1/detection_loss‚2.0057183274918278 .loss/FBt1/detection_loss‚2.605142280587862‡Sð 2loss/SRqqt1/assignment_loss‚1.6083147567089484 1loss/SRqqt2/assignment_loss‚1.340362674803259 1loss/SRqqt1/detection_loss‚2.0057183274918278 .loss/FBt1/detection_loss‚2.605142280587862 /loss/FRt1/detection_loss‚0.6175986351959097 /loss/FBt2/detection_loss‚0.9443253654186333 trainer/global_step‚699 % _timestamp‚1.7843683777516398e+09 1loss/FBt2/assignment_loss‚0.23073485293228996 1loss/SRqqt2/detection_loss‚0.6560458942140448 &loss/total_loss‚17.959855183399974 epoch‚0 0loss/FRt2/assignment_loss‚2.4110897795819657 1loss/FBt1/assignment_loss‚0.26586317923479796 /loss/FRt2/detection_loss‚1.3179435724131308 _runtime‚ 319.195177202 _step‚55 0loss/FRt1/assignment_loss‚3.95671586481730137¬ºXR‘ñ‚(:† „šíÒ¥öÒŠ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚181.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðZ˜ªò‚(:Ÿ „šíÒ±±öŒ disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 12.05078125 proc.cpu.threads‚17 disk.nvme0n1.in‚468.4375 disk.sda2.in‚0 disk.sda.out‚ 12.05078125 disk.nvme0c0n1.in‚468.4375 cpu‚26.66604913963469 network.sent‚701741 disk.sda2.out‚0" disk.dm-0.out‚1910.14501953125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚1910.14501953125 proc.memory.rssMB‚ 2604.3125 network.recv‚146115% disk.nvme0n1.out‚1910.14501953125 disk.sda.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚590.0056762695312 disk.dm-1.in‚0* proc.memory.percent‚15.895462036132812 disk.sda1.out‚0 disk.dm-1.out‚ 12.05078125 disk.sda3.in‚0$ memory_percent‚95.49829959869385* disk./.usagePercent‚39.595775982184165 disk.rbd0.out‚0 disk.dm-0.in‚468.4375 disk.rbd1.in‚0 disk.rbd1.out‚0.75( proc.memory.availableMB‚ 737.55859375„½ò0ó+ _wandbruntime‚341  _runtime‚341»Q(Ãô½ & lr-AdamW/pg1‚0.0007930055903209168 & lr-AdamW/pg2‚0.0007930055903209168 trainer/global_step‚749 $ _timestamp‚1.784368399727562e+09 _runtime‚ 341.171357716 _step‚56¥–ƒxÇõÁ & lr-AdamW/pg1‚0.0007930055903209168 & lr-AdamW/pg2‚0.0007930055903209168 trainer/global_step‚749 $ _timestamp‚1.784368399727562e+09 _runtime‚ 341.171357716 _step‚5684ûå0ö+ _wandbruntime‚341  _runtime‚341$ø:q§'÷¡O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26642745888768 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28269628940288 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27841929957376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚ 341.172686135 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27841946996736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27841929957376 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 443411653120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚57 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27363136264192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5419699 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3364445 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6995003 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 427786711552 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 427698982912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5419697 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27789301923180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12414702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26199143305216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166026752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5419697 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5419699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27789405097766 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 443577679872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1663319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104768000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26642745888768 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28269628940288 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27841929957376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚ 341.172686135 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11747 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27841946996736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27841929957376 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 443411653120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚57 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27363136264192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5419699 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3364445 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6995003 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 427786711552 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 427698982912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5419697 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27789301923180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12414702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26199143305216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166026752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5419697 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5419699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27789405097766 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 443577679872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1663319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12431218 _runtime‚ 341.511660242 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1665317 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28307309263872 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 444170943488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5034477 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚58 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27826341280038 % _timestamp‚1.7843684000676067e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27826444454736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6994714 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3368932 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27879040921600 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5426920 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 426732980518 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12421632 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26678053452288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26234079723520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27879057960960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚749 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 428268342272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12431218 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5426918 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28307414039040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5034246 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27879057960960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28307414039040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27399608299520 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7004298 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5426918 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3368929 ]JDeviceStatsMonitor.on_train_batch_end8ò„ÈÖ/requested_bytes.large_pool.allocated‚27399625338880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26678250667008 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936¦è¾lµ&û¯M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28307309263872 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 444170943488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5034477 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚58 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27826341280038 % _timestamp‚1.7843684000676067e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27826444454736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6994714 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3368932 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27879040921600 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5426920 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 426732980518 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12421632 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26678053452288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26234079723520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27879057960960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚749 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 428268342272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300167680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11747 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12431218 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5426918 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28307414039040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5034246 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27879057960960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28307414039040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27399608299520 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7004298 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5426918 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3368929 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27399625338880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11747 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26678250667008 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6994714 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 428356078080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12431218 _runtime‚ 341.511660242 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1665317 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880:TÜ#Ê0ü+ _wandbruntime‚341  _runtime‚3417š©ýþ 2loss/SRqqt2/assignment_loss‚1.4089198869625592 1loss/FBt1/assignment_loss‚0.09926556566833182 /loss/FRt1/detection_loss‚0.5969383896663203 2loss/SRqqt1/assignment_loss‚1.5870835319993661 /loss/FRt2/detection_loss‚1.3806995125457975 $ _timestamp‚1.784368400069288e+09 /loss/FBt1/detection_loss‚1.5949353055598645 &loss/total_loss‚16.362304126467798 0loss/FRt1/assignment_loss‚3.9351375376184783 1loss/FBt2/assignment_loss‚0.30640668612571237 1loss/SRqqt1/detection_loss‚1.9811845036580207 1loss/SRqqt2/detection_loss‚0.6428837076667907 trainer/global_step‚749 _step‚59 epoch‚0 _runtime‚ 341.512822287 /loss/FBt2/detection_loss‚0.5998151526921958 0loss/FRt2/assignment_loss‚2.2290343463043625»MBðþ‚ /loss/FBt2/detection_loss‚0.5998151526921958 0loss/FRt2/assignment_loss‚2.2290343463043625 2loss/SRqqt2/assignment_loss‚1.4089198869625592 1loss/FBt1/assignment_loss‚0.09926556566833182 /loss/FRt1/detection_loss‚0.5969383896663203 2loss/SRqqt1/assignment_loss‚1.5870835319993661 /loss/FRt2/detection_loss‚1.3806995125457975 $ _timestamp‚1.784368400069288e+09 /loss/FBt1/detection_loss‚1.5949353055598645 &loss/total_loss‚16.362304126467798 0loss/FRt1/assignment_loss‚3.9351375376184783 1loss/FBt2/assignment_loss‚0.30640668612571237 1loss/SRqqt1/detection_loss‚1.9811845036580207 1loss/SRqqt2/detection_loss‚0.6428837076667907 trainer/global_step‚749 _step‚59 epoch‚0 _runtime‚ 341.512822287;‘W¦œÿ‚(:‘ “šíÒí¾¢‹ gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚193.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.613400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ”ÿ¦€‚(:› “šíÒäÄ$ disk.nvme0n1.out‚1913.1669921875# disk./.usageGB‚590.005672454834 disk.rbd1.out‚0.75 disk.sda3.in‚0) disk./.usagePercent‚39.59577572617663! disk.dm-0.out‚1913.1669921875& disk.nvme0c0n1.out‚1913.1669921875 disk.sda3.out‚ 12.37890625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚468.4375 proc.memory.rssMB‚ 2604.3125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda1.in‚0 network.recv‚151368 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 12.37890625 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚15.895462036132812 network.sent‚749182 disk.rbd1.in‚0 disk.nvme0c0n1.in‚468.4375$ memory_percent‚95.49977779388428( proc.memory.availableMB‚ 737.31640625 cpu‚26.697210427236104 proc.cpu.threads‚17 disk.sda.out‚ 12.37890625 disk.sda2.out‚0 disk.dm-0.in‚468.4375ã8‘‚(:† ¢šíÒÙº©‹ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚173.397' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yo{²©‚‚(:ž ¢šíÒøÖÊŒ' disk.nvme0c0n1.out‚1918.79638671875$ memory_percent‚95.49899101257324 disk.rbd1.in‚0 disk.nvme0c0n1.in‚468.4375 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.sent‚796384 disk.nvme0n1.in‚468.4375 disk.rbd0.in‚0 network.recv‚156327( disk./.usagePercent‚39.5957739341239 disk.dm-0.in‚468.4375% disk.nvme0n1.out‚1918.79638671875 disk.sda2.out‚0* proc.memory.percent‚15.895462036132812' proc.memory.availableMB‚ 737.4453125" disk.dm-0.out‚1918.79638671875 disk.rbd1.out‚0.8125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.out‚ 12.58984375$ disk./.usageGB‚590.0056457519531 disk.dm-1.in‚0 disk.sda3.out‚ 12.58984375 disk.sda.in‚0 cpu‚26.72934416830709 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2604.3125 disk.dm-1.out‚ 12.58984375 disk.sda3.in‚0„U Ó0ƒ+ _wandbruntime‚363  _runtime‚363I+ozÄ„¾ & lr-AdamW/pg1‚0.0008459432131727804 & lr-AdamW/pg2‚0.0008459432131727804 trainer/global_step‚799 % _timestamp‚1.7843684219595191e+09 _runtime‚ 363.403289247 _step‚60zÌÈ…Â & lr-AdamW/pg1‚0.0008459432131727804 & lr-AdamW/pg2‚0.0008459432131727804 trainer/global_step‚799 % _timestamp‚1.7843684219595191e+09 _runtime‚ 363.403289247 _step‚60<4uLÄ0†+ _wandbruntime‚363  _runtime‚363x ‹¼§'‡¡O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300167680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015666743 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5780747 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29697478168576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 456199207936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103174586 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29186755069952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚ 363.404677328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28417868570112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29641301669288 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7460905 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27944719036416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30153677376512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29697495207936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 456286936576 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7451335 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5780749 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5363320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473124630016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30153677376512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3588645 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197221888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29697495207936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5363090 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚61 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1774672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29186738030592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 454563638696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28417671348224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30153782144512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29697478168576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13232082 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172318208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30153782144512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13241654 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5780747 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29641404843874 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 456199207936 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30153677376512 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3588645 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197221888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11694 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29697495207936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87728640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87728640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5363090 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚61 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9570 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1774672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29186738030592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 454563638696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28417671348224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30153782144512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29697478168576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86135226 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13232082 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172318208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298570279 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30153782144512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104768000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081461760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13241654 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5780747 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29641404843874 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 456199207936 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081461760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28417868570112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29641301669288 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7460905 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27944719036416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30153677376512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29697495207936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9572 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11747 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 456286936576 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7451335 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5780749 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300167680 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5363320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473124630016=äxü0‰+ _wandbruntime‚363  _runtime‚363½.<í±&Š«M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13248584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29734606172160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 456768567296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29734606172160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 363.741920498 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7470200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 473711356416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1776867 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚62 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7470200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5787970 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 456856303104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28453366811136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13258170 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1776647 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27979630551040 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29678341026146 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5787968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3593129 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193020416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7460616 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 455217095564 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30191462475264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29734589132800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29223210065920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13248584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30191357700096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30191462475264 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28453173790720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29223227105280 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5369776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27979655454720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13258170 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5369999 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5787970Vjñµ&‹¯M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5787970 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 456856303104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103174698 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28453366811136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13258170 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081461760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1776647 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27979630551040 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29678341026146 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5787968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3593129 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193020416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7460616 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081461760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298570279 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104775168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87735808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 455217095564 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30191462475264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29734589132800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29223210065920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9586 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13248584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104775168 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30191357700096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30191462475264 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28453173790720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015666743 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29223227105280 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5369776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27979655454720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300167680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13258170 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5369999 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86135338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87735808 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5787970 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13248584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29734606172160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 456768567296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29734606172160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 363.741920498 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7470200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 473711356416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1776867 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚62 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7470200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146>T4”ë0Œ+ _wandbruntime‚363  _runtime‚3637ù½ú epoch‚0 trainer/global_step‚799 $ _timestamp‚1.784368422299444e+09 1loss/FBt1/assignment_loss‚0.10582666364166471 /loss/FRt1/detection_loss‚0.6295559038171632 1loss/SRqqt1/detection_loss‚1.9609000457797285 _runtime‚ 363.742986301 /loss/FRt2/detection_loss‚1.4096353904824823 .loss/FBt1/detection_loss‚2.131432829932733 /loss/FBt2/detection_loss‚0.7418874441880065 2loss/SRqqt1/assignment_loss‚1.3740568160577569 1loss/SRqqt2/assignment_loss‚1.246409775387673 1loss/FBt2/assignment_loss‚0.12695198256163628 _step‚63 /loss/FRt1/assignment_loss‚3.755175305530429 /loss/FRt2/assignment_loss‚1.944126670975839 1loss/SRqqt2/detection_loss‚0.6230526260593684 &loss/total_loss‚16.049011454414483ÀåGŽþ /loss/FBt2/detection_loss‚0.7418874441880065 2loss/SRqqt1/assignment_loss‚1.3740568160577569 1loss/SRqqt2/assignment_loss‚1.246409775387673 1loss/FBt2/assignment_loss‚0.12695198256163628 _step‚63 /loss/FRt1/assignment_loss‚3.755175305530429 /loss/FRt2/assignment_loss‚1.944126670975839 1loss/SRqqt2/detection_loss‚0.6230526260593684 &loss/total_loss‚16.049011454414483 epoch‚0 trainer/global_step‚799 $ _timestamp‚1.784368422299444e+09 1loss/FBt1/assignment_loss‚0.10582666364166471 /loss/FRt1/detection_loss‚0.6295559038171632 1loss/SRqqt1/detection_loss‚1.9609000457797285 _runtime‚ 363.742986301 /loss/FRt2/detection_loss‚1.4096353904824823 .loss/FBt1/detection_loss‚2.131432829932733?Öéï‘‚(:† ±šíÒááý‹ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚191.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.3072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0量ԭ‚(:¢ ±šíÒ®¦óŒ network.recv‚160315$ disk./.usageGB‚590.0056495666504 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 12.7109375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚468.4375% disk.nvme0n1.out‚1921.67724609375 disk.rbd0.in‚0 disk.dm-1.out‚ 12.7109375 disk.drbd2165.in‚0 cpu‚26.864172582890777 network.sent‚801830 disk.rbd1.in‚0" disk.dm-0.out‚1921.67724609375 disk.rbd1.out‚0.8125 disk.nvme0n1.in‚468.4375$ memory_percent‚95.74086666107178( proc.memory.availableMB‚ 697.81640625# proc.memory.rssMB‚ 2643.40234375' disk.nvme0c0n1.out‚1921.67724609375 disk.sda2.in‚0) disk./.usagePercent‚39.59577419013143 disk.rbd0.out‚0 disk.dm-0.in‚468.4375 disk.sda3.in‚0 disk.sda3.out‚ 12.7109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.134047508239746¦“J‘bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPVñÕ-J’bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPºõJ“bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP鮩§J”bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP@UɶH•bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP‡˜bkJ–bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÜWœ=H—bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÞBgk@˜b; particle/accuracy_1_of_4"trainer/global_step2:JP`/¾H™bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP]Y HšbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP~=çH›bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP<%é·NœbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP {ŒHbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP͇DJžbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¹SHŸbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP r*ÕJ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP¯œMIN¡bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP(O;†@¢b; particle/accuracy_3_of_5"trainer/global_step2:JPêFä@£b; particle/accuracy_4_of_5"trainer/global_step2:JP_¬,•J¤bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPµkW‡J¥bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPŠ@¦b; particle/accuracy_2_of_3"trainer/global_step2:JPÂ÷´J§bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPáÑGR@¨b; particle/accuracy_1_of_5"trainer/global_step2:JP_ ¤2H©bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP€váJªbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP±³âN«bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP´:kr@¬b; particle/accuracy_3_of_4"trainer/global_step2:JP9|ÓN­bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP‡=ýN®bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP›Ù&‡N¯bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPa´$N°bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPì¶NrJ±bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPM\<@²b; particle/accuracy_2_of_2"trainer/global_step2:JPÌÇDN³bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPL¹0J´bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP ã/HµbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPNŒr•J¶bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP‡Ê€N·bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÓl\5H¸bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPµmãH¹bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPìƉJºbE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPx³P$J»bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP­ã‘J¼bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPU~`N½bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPyÙšN¾bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP2^´…J¿bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP`í ›JÀbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPH“ÒHÁbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP·NZ^NÂbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP(xÓ8Ãb3 particle/f_score"trainer/global_step2:JPñ-”HÄbC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPR^ïNÅbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPðPÀ¢HÆbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP•6°pHÇbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPqâ–@Èb; particle/accuracy_1_of_2"trainer/global_step2:JP­c0HÉbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÍ&ã±NÊbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP¼âLžHËbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP½tJJÌbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP™NÍbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP~poWNÎbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP”É­HÏbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP:ˆ[HÐbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPHoï HÑbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÜ»;Òb6 jet/accuracy_5_of_5"trainer/global_step2:JP¼=;rJÓbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPƒôøFJÔbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP"ñƒJÕbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÁÚâHÖbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP%äºN×bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP‡ÊVNØbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPCT{êJÙbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP0¦ËÊ;Úb6 loss/val_total_loss"trainer/global_step2:JP8,ˆJÛbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPSŒÎ›NÜbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPøÙ”.JÝbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPdG|NÞbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP"¥ÌHßbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÀ@HàbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPvœÏdNábI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPBOª´HâbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPçq§ÑNãbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPðÄ3`JäbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPwˆëWJåbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÅ_:;æbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/globznã/al_step2:JPÚÞJJçbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP'ÚnG;èb6 jet/accuracy_4_of_4"trainer/global_step2:JPù{½@éb; particle/accuracy_1_of_1"trainer/global_step2:JP7 ÂHêbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPUÁîÌHëbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP_£$JìbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP>ÍzoHíbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP4}NîbI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP›HvŠJïbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP)À½…HðbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPb»¹JñbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPè…JòbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP? *|JóbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPåäÔ‰HôbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPƒº·HõbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP~2³HöbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÀîÌ™N÷bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP (Z HøbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÀôJùbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP€úUJúbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP{ d@ûb; particle/accuracy_5_of_5"trainer/global_step2:JP[] HübC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP1Í„ÐJýbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPMËTHþbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPä’E‰JÿbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP $‰N€bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÉ#hº;b6 jet/accuracy_2_of_4"trainer/global_step2:JPäL³N‚bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP~#àHƒbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÇcÚÁH„bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP%i“J…bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\„<ûH†bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP(Ö¦cN‡bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPë•ÚKJˆbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP‚'; N‰bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPØ3ì%NŠbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP›£âÝH‹bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÊH—4JŒbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPøΰ@b; particle/accuracy_4_of_4"trainer/global_step2:JPèCJŽbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP`—åßHbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPó{ÀUNbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPýx±2;‘b6 jet/accuracy_2_of_3"trainer/global_step2:JPÂB­J’bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPËx‚íJ“bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP˜S€[;”b6 jet/accuracy_1_of_3"trainer/global_step2:JP›%à/@•b; particle/accuracy_2_of_4"trainer/global_step2:JPX½ˆ;–b6 jet/accuracy_3_of_5"trainer/global_step2:JP˜•…N—bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP\…ö·H˜bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP^ŽH™bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÃõøÒJšbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP/¼)WH›bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPd.^NœbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPºÖ?SJbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP—£øÞJžbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP4ÀÐHŸbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP=8vN bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÂQÏÛJ¡bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPŸ»ÍŽN¢bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP"áòH£bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP+b5;¤b6 jet/accuracy_2_of_2"trainer/global_step2:JP¢«“H¥bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP?Á =H¦bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPh"ÚN§bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPAŽùñN¨bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP|—J©bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPe¡“HªbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPØR J«bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPtfN¬bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPlø¼‚J­bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÇÊ;®b6 jet/accuracy_4_of_5"trainer/global_step2:JP¥÷-MH¯bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPªêÃH°bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPQàòíN±bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÀwœ>N²bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPŸ*ñîJ³bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÐôÂlN´bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¨ôÖJµbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPçìJ¶bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÝǼ@·b; particle/accuracy_2_of_5"trainer/global_step2:JP7ƒÌJ¸bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP-\qÀJ¹bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPœ´H±HºbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP‡|FHH»bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÛ›N¼bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPTõVJ½bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP={¿]H¾bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP›–ÅúN¿bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPN_'ìJÀbE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPøÒ˜GHÁbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP¶3 àJÂbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPƒ-øHÃbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÒcJÆHÄbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPsSJÅbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP⩆#JÆbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPx—úHÇbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP(n1JÈbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÅÐ=NÉbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP>¸´¾JÊbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]oÛlJËbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP—›bHÌbC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPîp“âHÍbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPOh´<Îb7 particle/sensitivity"trainer/global_step2:JPÒ´hyGÏbB validation_average_jet_accuracy"trainer/global_step2:JP((²ÑHÐbC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÉc´ôJÑbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPîâNÒbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPëØœï;Ób6 jet/accuracy_3_of_3"trainer/global_step2:JPpŸ½ÑNÔbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPïASóJÕbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP_r÷ƒJÖbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÁzaaJ×bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP‹¤JØbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP0hXmNÙbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPGç`';Úb6 jet/accuracy_1_of_2"trainer/global_step2:JP ,¸NÛbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPiï1»JÜbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP‘¶NÝbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP,‹º;Þb6 jet/accuracy_1_of_1"trainer/global_step2:JP[gEb;ßb6 jet/accuracy_1_of_5"trainer/global_step2:JPóšÚHàbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP­Žb©NábI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP—¶é€JâbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPó»áÛJãbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP_G’HäbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÄÌW‡HåbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP.¦«xHæbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP8j UNçbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP«@ésNèbI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¨a" JébE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÃUð¨NêbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPb_)HëbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPáw¯™JìbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP\a ÅNíbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPxsõJîbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPë¥0JïbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP°¦÷!;ðb6 jet/accuracy_3_of_4"trainer/global_step2:JPræqHñbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPL…,9NòbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP+ey@ób; particle/accuracy_3_of_3"trainer/global_step2:JP"Ä‘»;ôb6 jet/accuracy_2_of_5"trainer/global_step2:JPÐ#‰.JõbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPééR JöbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP/ u…J÷bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÒþJøbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP%ØZµHùbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPL†HúbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP†*[´JûbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP±íHübC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP ¯dJýbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÜHþbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP­1NÿbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPµ…¬J€bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¹9åHbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÊ.æH‚bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÍRðïJƒbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP¿vkJ„bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÅ2›J…bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP0,اN†bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPiÈ:J‡bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPôò—RJˆbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPãà8àH‰bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP}x^aHŠbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPìxÞJ‹bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP%sÂÎHŒbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP°C¿JbE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÞÆíðHŽbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPîD;b6 jet/accuracy_1_of_4"trainer/global_step2:JPÞo¦Ë@b; particle/accuracy_1_of_3"trainer/global_step2:JPÿ®J‘bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP+–”HJ’bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPOÛÄH“bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP%ƒ]þH”bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPwîÂN•bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP€—ËN–bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPcTè„J—bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP1 öJ˜bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP„«,ŽH™bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP?âNšbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPñ¦fH›bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÔb `JœbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPF¬U6JbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŽ#H,JžbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP’ÉÕ9Ÿb4 particle/accuracy"trainer/global_step2:JPŒÆ•< b7 particle/specificity"trainer/global_step2:JP4IñH¡bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP‰ç«¶J¢bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP§/ÒJ£bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPýaäkH¤bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÛ€VJ¥bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPBXéýN¦bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPˆÑq+N§bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP>ñUH¨bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPï¸H©bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPD‹$HªbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPY$ïN«bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÎ`ÌH¬bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÞâɺN­bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¶{ïˆJ®bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPàM3H¯bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPw©†PJ°bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPV÷ŽÎ0±+ _wandbruntime‚384  _runtime‚384¿Néà±=²«{ =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9920051097869873 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.22249959409236908 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4524114429950714 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9904158711433411 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0814451351761818 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.11985328048467636 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6800000071525574 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.356259286403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14614006876945496 )loss/val_total_loss‚16.64988018274758 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6458185911178589 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05351276695728302 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16178244352340698 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3499999940395355 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9804667234420776 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _runtime‚ 384.421773689 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8416054844856262 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06102290749549866 jet/accuracy_4_of_4‚0 particle/accuracy_1_of_1‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9924588799476624 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7523753046989441 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.399958074092865 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4390977621078491 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0555555559694767 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09126987308263779 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03109101951122284 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9956207275390625 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23567567765712738 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4435897171497345 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.31597134470939636 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07058033347129822 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.17878542840480804 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9583863019943237 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37345677614212036 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_2_of_4‚0.2532784640789032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9915175437927246 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9946145415306091 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.22602973878383636 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.426754355430603 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.15739214420318604 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.18617337942123413 /particle/accuracy_4_of_4‚0.8981267809867859 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3854166567325592 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 +jet/accuracy_2_of_3‚0.22300851345062256 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07515383511781693 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.6663408875465393 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08553978055715561 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.331623911857605 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4839654564857483 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.1350979059934616 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.3114677965641022 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7991639971733093 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.07085107266902924 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9903846383094788 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9595965147018433 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.16958977282047272 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9438797235488892 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2469174712896347 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.005169851239770651 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16490322351455688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5222857594490051 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07248679548501968 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8392857313156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.166513592004776 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952380657196045 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23750989139080048 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3772779107093811 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3571428656578064 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.15739214420318604 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00986825954169035 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.225199356675148 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2654446065425873 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4345352053642273 +particle/sensitivity‚0.5059008002281189 7validation_average_jet_accuracy‚0.16696889698505402 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.16576097905635834 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_3_of_3‚0.030326902866363525 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.18617337942123413 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.3578946888446808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚800 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_2‚0.4770951569080353 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8939386010169983 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_1_of_1‚0.12230021506547928 jet/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08220358192920685 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.18005864322185516 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003117914078757167 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.977135181427002 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929776787757874 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3759439289569855 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5333822965621948 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01406249962747097 ,jet/accuracy_3_of_4‚0.043479274958372116 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.92083340883255 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_3‚0.9989849328994751 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49760863184928894 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004776536487042904 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.06666667014360428 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9723644256591797 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4078008234500885 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4096190929412842 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01366420742124319 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.29292377829551697 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.17466473579406738 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9777874946594238 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.15236712992191315 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6800000071525574 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.030112793669104576 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.14634615182876587 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15409211814403534 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08246929198503494 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09885967522859573 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.1647547483444214 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2571720480918884 *jet/accuracy_1_of_4‚0.6546899080276489 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01339147612452507 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.18578359484672546 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9864159822463989 5 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16252401471138 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843684429772737e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06943928450345993 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9944870471954346 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9819897413253784 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.17131593823432922 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.7996573448181152 +particle/specificity‚0.8800022602081299 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.1480294167995453 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.23361755907535553 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.41517308354377747 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0023583206348121166 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42813801765441895 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9893426299095154 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.29065102338790894 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 _step‚64 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.511432945728302 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7028098702430725 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4148173928260803 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2239753156900406 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.33202409744262695 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15447552502155304 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.15659679472446442 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16238173842430115 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07228468358516693 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2571720480918884 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9832630157470703 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8406630158424377 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47213953733444214 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07248679548501968 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.06666667014360428 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9855164289474487 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.14442561566829681 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 /particle/accuracy_3_of_4‚0.9922618865966797 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.17884176969528198 particle/accuracy_2_of_2‚1 epoch‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15139533579349518 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11948555707931519 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.35810238122940063 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38257578015327454 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012971495743840933 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9915175437927246 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2942950427532196 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.5202611088752747 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9469692707061768 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35234373807907104 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24340149760246277 particle/accuracy_1_of_2‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9868420958518982š¥ƒ&³¯{ @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.16958977282047272 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9438797235488892 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2469174712896347 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.005169851239770651 9"Purity/0FRt1SRqqt2FBÏ2£ :t/SRqqt_purity‚0.1428571492433548 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16490322351455688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5222857594490051 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07248679548501968 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8392857313156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.166513592004776 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9952380657196045 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23750989139080048 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3772779107093811 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3571428656578064 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.15739214420318604 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00986825954169035 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.225199356675148 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2654446065425873 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4345352053642273 +particle/sensitivity‚0.5059008002281189 7validation_average_jet_accuracy‚0.16696889698505402 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.16576097905635834 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_3_of_3‚0.030326902866363525 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.18617337942123413 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.3578946888446808 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚800 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_2‚0.4770951569080353 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8939386010169983 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_1_of_1‚0.12230021506547928 jet/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08220358192920685 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.18005864322185516 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003117914078757167 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.977135181427002 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929776787757874 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3759439289569855 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5333822965621948 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01406249962747097 ,jet/accuracy_3_of_4‚0.043479274958372116 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.92083340883255 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_3‚0.9989849328994751 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49760863184928894 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004776536487042904 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.06666667014360428 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9723644256591797 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4078008234500885 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4096190929412842 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01366420742124319 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.29292377829551697 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.17466473579406738 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9777874946594238 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.15236712992191315 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6800000071525574 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.030112793669104576 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.14634615182876587 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15409211814403534 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08246929198503494 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09885967522859573 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.1647547483444214 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2571720480918884 *jet/accuracy_1_of_4‚0.6546899080276489 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01339147612452507 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.18578359484672546 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9864159822463989 5 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16252401471138 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843684429772737e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06943928450345993 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9944870471954346 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9819897413253784 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.17131593823432922 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.7996573448181152 +particle/specificity‚0.8800022602081299 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.1480294167995453 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.23361755907535553 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.41517308354377747 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0023583206348121166 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42813801765441895 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9893426299095154 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.29065102338790894 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 _step‚64 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.511432945728302 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7028098702430725 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4148173928260803 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2239753156900406 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.33202409744262695 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15447552502155304 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.15659679472446442 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16238173842430115 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07228468358516693 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2571720480918884 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9832630157470703 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8406630158424377 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47213953733444214 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07248679548501968 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.06666667014360428 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9855164289474487 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.14442561566829681 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 /particle/accuracy_3_of_4‚0.9922618865966797 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.17884176969528198 particle/accuracy_2_of_2‚1 epoch‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15139533579349518 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11948555707931519 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.35810238122940063 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38257578015327454 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012971495743840933 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9915175437927246 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2942950427532196 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.5202611088752747 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9469692707061768 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35234373807907104 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24340149760246277 particle/accuracy_1_of_2‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9868420958518982 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9920051097869873 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.22249959409236908 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4524114429950714 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9904158711433411 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0814451351761818 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.11985328048467636 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6800000071525574 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.356259286403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14614006876945496 )loss/val_total_loss‚16.64988018274758 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6458185911178589 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05351276695728302 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16178244352340698 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3499999940395355 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9804667234420776 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _runtime‚ 384.421773689 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8416054844856262 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06102290749549866 jet/accuracy_4_of_4‚0 particle/accuracy_1_of_1‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9924588799476624 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7523753046989441 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.399958074092865 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4390977621078491 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0555555559694767 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09126987308263779 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03109101951122284 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9956207275390625 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23567567765712738 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4435897171497345 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.31597134470939636 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07058033347129822 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.17878542840480804 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9583863019943237 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37345677614212036 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_2_of_4‚0.2532784640789032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9915175437927246 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9946145415306091 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.22602973878383636 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.426754355430603 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.15739214420318604 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.18617337942123413 /particle/accuracy_4_of_4‚0.8981267809867859 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3854166567325592 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 +jet/accuracy_2_of_3‚0.22300851345062256 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07515383511781693 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.6663408875465393 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08553978055715561 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.331623911857605 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4839654564857483 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.1350979059934616 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.3114677965641022 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7991639971733093 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.07085107266902924 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9903846383094788 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9595965147018433@;ìûbœ´‚(:‘ ÀšíÒ¹¡‘‹ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚129.709' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.941799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Þu8±µ‚(:¦ ÀšíÒ±€‘% disk.nvme0n1.out‚1974.35693359375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 535.1953125) disk./.usagePercent‚39.60418147755588 disk.rbd1.out‚0.875 disk.nvme0n1.in‚ 535.1953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 13.2578125 disk.sda3.out‚ 13.2578125 network.sent‚848503% proc.memory.availableMB‚ 567.34375 disk.rbd0.out‚0 network.recv‚164843 disk.sda.in‚0 disk.sda.out‚ 13.2578125* proc.memory.percent‚16.155362129211426 cpu‚26.942043464336294 disk.drbd2165.in‚0# memory_percent‚96.5372085571289 disk.rbd1.in‚0' disk.nvme0c0n1.out‚1974.35693359375 disk.rbd0.in‚0" disk.dm-0.out‚1974.35693359375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2646.89453125$ disk./.usageGB‚590.1309242248535 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 535.195312573û‘¶‚(:† ÏšíÒîûÄŠ gpu.0.gpu‚18.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ò¸­·‚(:¢ ÏšíÒ‹£¶Œ disk.sda2.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.60419248587985 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 535.1953125# proc.memory.rssMB‚ 2646.90234375* proc.memory.percent‚16.155409812927246 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 13.421875 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 535.1953125$ disk.nvme0n1.out‚2058.6181640625 disk.rbd0.in‚0 disk.sda3.out‚ 13.421875 disk.sda2.in‚0 disk.dm-0.in‚ 535.1953125! disk.dm-0.out‚2058.6181640625 cpu‚26.91872102587924 proc.cpu.threads‚17 disk.dm-1.out‚ 13.421875$ memory_percent‚96.53925895690918' proc.memory.availableMB‚ 567.0078125 network.sent‚906066 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.875& disk.nvme0c0n1.out‚2058.6181640625 network.recv‚170503$ disk./.usageGB‚590.1310882568359 disk.sda3.in‚0 disk.sda1.out‚0#Íù0¸+ _wandbruntime‚406  _runtime‚406 ŠtĹ¾ & lr-AdamW/pg2‚0.0008480575884679996 trainer/global_step‚849 % _timestamp‚1.7843684651894693e+09 _runtime‚ 406.633210048 _step‚65 & lr-AdamW/pg1‚0.0008480575884679996ÿ>ÈȺ & lr-AdamW/pg1‚0.0008480575884679996 & lr-AdamW/pg2‚0.0008480575884679996 trainer/global_step‚849 % _timestamp‚1.7843684651894693e+09 _runtime‚ 406.633210048 _step‚65AŒFB0»+ _wandbruntime‚406  _runtime‚406[¼‘K§'¼¡O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 501301075968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32514005733376 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33015323848704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5868266 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31967557670912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 501213139968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8070496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1951063 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚849 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30604808192000 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6376327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6376325 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30604833095680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6376325 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32514022772736 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8070496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3917425 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32514005733376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32467111066039 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 499450218685 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 501301075968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 499536355767 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 % _timestamp‚1.7843684651904478e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33015218873344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33015218873344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8060521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376(ö;½¥O ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31124048578560 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 519039177728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31967574710272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 519215482880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚ 406.634503641 _step‚66 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 501213139968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3917428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201208832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32467007889597 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33015323848704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32514022772736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14446823 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31123847369728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6376327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33015218873344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33015218873344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8060521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 501301075968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32514005733376 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33015323848704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5868266 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319675576709«¥12 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 501213139968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8070496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1951063 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚849 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30604808192000 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6376327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6376325 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30604833095680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6376325 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32514022772736 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8070496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3917425 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32514005733376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32467111066039 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 499450218685 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 501301075968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 499536355767 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 % _timestamp‚1.7843684651904478e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3921909 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33053004179456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14463339 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32551133736960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32504047246455 _step‚67 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31159343393792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8079791 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 501782499328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6383548 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33052899196928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5874946 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 501782499328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205395968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8069802 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32004029706240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6383546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6383548 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6383546 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33053004179456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 500017540215 $ _timestamp‚1.784368465528221e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32551116697600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32551116697600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1953271 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 501870442496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3921912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473Ú½ñQ´&À®M EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14463339 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32551133736960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32504047246455 _step‚67 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31159343393792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8079791 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 501782499328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6383548 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33052899196928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5874946 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 501782499328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205395968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8069802 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32004029706240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6383546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6383548 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6383546 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33053004179456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 500017540215 $ _timestamp‚1.784368465528221e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32551116697600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32551116697600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1953271 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 501870442496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3921912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 501870442496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32004046745600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8069802 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5875183 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 519804179456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚ 406.972229549 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32551133736960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30639719706624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180492288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3921909 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33053004179456C‡fŠ0Á+ _wandbruntime‚406  _runtime‚406ZÏ¥ýÂ÷ /loss/FRt2/assignment_loss‚2.124697661893383 1loss/FBt2/assignment_loss‚0.05726064139118053 /loss/FRt2/detection_loss‚1.3856893870267708 1loss/SRqqt1/detection_loss‚1.7805226973186319 /loss/FBt1/detection_loss‚1.5773139614134262 %loss/total_loss‚16.07951053392846 _runtime‚ 406.973924299 1loss/SRqqt1/assignment_loss‚1.729675350137628 1loss/SRqqt2/assignment_loss‚1.157939463964332 1loss/SRqqt2/detection_loss‚0.6012143964719109 trainer/global_step‚849 0loss/FBt1/assignment_loss‚0.1369390219437649 /loss/FRt1/detection_loss‚0.5786042662468426 /loss/FBt2/detection_loss‚1.3155088257756242 epoch‚1 # _timestamp‚1.78436846553039e+09 _step‚68 /loss/FRt1/assignment_loss‚3.634144860344964çjö<Ãû 0loss/FBt1/assignment_loss‚0.1369390219437649 /loss/FRt1/detection_loss‚0.5786042662468426 /loss/FBt2/detection_loss‚1.3155088257756242 epoch‚1 # _timestamp‚1.78436846553039e+09 _step‚68 /loss/FRt1/assignment_loss‚3.634144860344964 /loss/FRt2/assignment_loss‚2.124697661893383 1loss/FBt2/assignment_loss‚0.05726064139118053 /loss/FRt2/detection_loss‚1.3856893870267708 1loss/SRqqt1/detection_loss‚1.7805226973186319 /loss/FBt1/detection_loss‚1.5773139614134262 %loss/total_loss‚16.07951053392846 _runtime‚ 406.973924299 1loss/SRqqt1/assignment_loss‚1.729675350137628 1loss/SRqqt2/assignment_loss‚1.157939463964332 1loss/SRqqt2/detection_loss‚0.6012143964719109 trainer/global_step‚849D@U{œÄ‚(:‘ ÞšíÒ‘Ó gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚175.529' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.105799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gN>f³Å‚(:¨ ÞšíÒÉðË disk.dm-1.out‚ 13.67578125! disk.nvme0c0n1.in‚ 535.1953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0! disk.dm-0.out‚2064.5283203125& disk.nvme0c0n1.out‚2064.5283203125* proc.memory.percent‚16.155409812927246 proc.cpu.threads‚17$ memory_percent‚96.53949737548828 disk.dm-0.in‚ 535.1953125 disk.sda.out‚ 13.67578125 disk.drbd2165.out‚0 disk.rbd1.out‚0.9375 cpu‚26.897056060063022 disk.sda3.in‚0 disk.sda3.out‚ 13.67578125 disk.sda.in‚0 network.sent‚911426) disk./.usagePercent‚39.60418557367643 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 535.1953125% proc.memory.availableMB‚ 566.96875# proc.memory.rssMB‚ 2646.90234375$ disk.nvme0n1.out‚2064.5283203125 network.recv‚174526$ disk./.usageGB‚590.1309852600098 disk.rbd0.in‚0bEÄ0Æ+ _wandbruntime‚428  _runtime‚428"âA™ÄǾ % _timestamp‚1.7843684867759988e+09 _runtime‚ 428.219798518 _step‚69 & lr-AdamW/pg1‚0.0008480476726257699 & lr-AdamW/pg2‚0.0008480476726257699 trainer/global_step‚899dî.ºÈÈ trainer/global_step‚899 % _timestamp‚1.7843684867759988e+09 _runtime‚ 428.219798518 _step‚69 & lr-AdamW/pg1‚0.0008480476726257699 & lr-AdamW/pg2‚0.0008480476726257699E²{qü0É+ _wandbruntime‚428  _runtime‚428¤´çëü Ê O [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33791159437312 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8536398 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34369570983936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8526423 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34319007635705 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6737375 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 _runtime‚ 428.221059318 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 529801300992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34319110812147 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4141625 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8536398 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 529713364992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32350408826880 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 548629659648 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34899267309568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6737377 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32350383923200 $ _timestamp‚1.784368486777063e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 529801300992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34319110812147 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4141625 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8536398 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 529713364992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32350408826880 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 548629659648 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34899267309568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6737377 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32350383923200 $ _timestamp‚1.784368486777063e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33827631472640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34356046992563 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15290291 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6744598 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 530370667520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4146112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15290291 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6208043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32934343773184 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 528415519923 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 530282724352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34406681948160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34406664908800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2061934 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34406681948160 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6208285 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34356150169117 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33827648512000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6744598 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 530282724352 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32934538683392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 528501657117 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 % _timestamp‚1.7843684871400347e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194910208 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 549218341888 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34936947633152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6744596 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8535704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32385295437824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34406664908800 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 428.584072538 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 % _timestamp‚1.7843684871400347e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194910208 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 549218341888 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34936947633152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6744596 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8535704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32385295437824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34406664908800 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 428.584072538 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33827631472640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34356046992563 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15290291 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6744598 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 530370667520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4146112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15290291 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6208043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32934343773184 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 528415519923 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 530282724352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34406681948160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34406664908800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2061934 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34406681948160 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6208285 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34356150169117 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33827648512000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6744598 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 530282724352 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32934538683392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 528501657117 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991G­FP0Ï+ _wandbruntime‚428  _runtime‚428+ýÝýÐ÷ 0loss/FRt1/assignment_loss‚3.7183876620992176 1loss/SRqqt2/detection_loss‚0.7513589938763445 &loss/total_loss‚15.700608713943334 epoch‚1 .loss/FRt2/detection_loss‚1.439276728348303 0loss/SRqqt1/detection_loss‚1.929994379161018 _runtime‚ 428.585149952 _step‚72 1loss/SRqqt1/assignment_loss‚1.361461255289061 0loss/FBt2/assignment_loss‚0.0712919345678074 /loss/FRt1/detection_loss‚0.5272791880086275 .loss/FBt1/detection_loss‚1.437001178200756 .loss/FBt2/detection_loss‚1.061863641407399 0loss/FRt2/assignment_loss‚2.1172794977306153 1loss/SRqqt2/assignment_loss‚1.151074231167944 0loss/FBt1/assignment_loss‚0.1343400240862398 trainer/global_step‚899 % _timestamp‚1.7843684871416168e+09¯¥}wÑû &loss/total_loss‚15.700608713943334 epoch‚1 .loss/FRt2/detection_loss‚1.439276728348303 0loss/SRqqt1/detection_loss‚1.929994379161018 _runtime‚ 428.585149952 _step‚72 1loss/SRqqt1/assignment_loss‚1.361461255289061 0loss/FBt2/assignment_loss‚0.0712919345678074 /loss/FRt1/detection_loss‚0.5272791880086275 .loss/FBt1/detection_loss‚1.437001178200756 .loss/FBt2/detection_loss‚1.061863641407399 0loss/FRt2/assignment_loss‚2.1172794977306153 1loss/SRqqt2/assignment_loss‚1.151074231167944 0loss/FBt1/assignment_loss‚0.1343400240862398 trainer/global_step‚899 % _timestamp‚1.7843684871416168e+09 0loss/FRt1/assignment_loss‚3.7183876620992176 1loss/SRqqt2/detection_loss‚0.7513589938763445HªM‚ÊÒ‚(:„ íšíÒÝ¡ ‹ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚186.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¨ÌžµÓ‚(:ª íšíҋΦ disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚0.9375' disk.nvme0c0n1.out‚2069.29345703125 disk.nvme0n1.in‚ 535.1953125" disk.dm-0.out‚2069.29345703125) disk./.usagePercent‚39.60417021322438 disk.drbd2165.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2646.90234375# memory_percent‚97.3231554031372 network.sent‚974376 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚180237 disk.sda.out‚ 13.9296875 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 535.1953125* proc.memory.percent‚16.155409812927246 cpu‚26.875794607629846 disk.dm-1.out‚ 13.9296875! disk.nvme0c0n1.in‚ 535.1953125% disk.nvme0n1.out‚2069.29345703125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 438.57421875$ disk./.usageGB‚590.1307563781738 disk.sda3.in‚0 disk.sda3.out‚ 13.9296875¸«ÔíÔ‚(:… üšíÒæô¦‹ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚171.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/U 3²Õ‚(:§ üšíÒ•ªà$ disk./.usageGB‚590.1306304931641 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.32258319854736 proc.cpu.threads‚17 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 2074.41015625 disk.nvme0n1.in‚ 607.94921875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2646.90234375 cpu‚26.8555682117894) disk./.usagePercent‚39.60416176497574 disk.drbd2165.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 607.94921875" disk.nvme0n1.out‚ 2074.41015625 disk.rbd0.out‚0 disk.dm-0.in‚ 607.94921875 network.sent‚ 1.037298e+06 disk.sda3.in‚0 disk.sda3.out‚ 14.35546875 disk.sda.out‚ 14.35546875 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 14.35546875 disk.dm-0.out‚ 2074.41015625* proc.memory.percent‚16.155409812927246 disk.sda1.in‚0( proc.memory.availableMB‚ 438.66796875 disk.rbd1.out‚1 disk.sda2.in‚0 network.recv‚185710”L0Ö+ _wandbruntime‚450  _runtime‚450œx*ÎÄ×¾ & lr-AdamW/pg1‚0.0008480309652856152 & lr-AdamW/pg2‚0.0008480309652856152 trainer/global_step‚949 % _timestamp‚1.7843685087751138e+09 _runtime‚ 450.218966026 _step‚73áè³þÈØ _step‚73 & lr-AdamW/pg1‚0.0008480309652856152 & lr-AdamW/pg2‚0.0008480309652856152 trainer/global_step‚949 % _timestamp‚1.7843685087751138e+09 _runtime‚ 450.218966026IDw+t0Ù+ _wandbruntime‚450  _runtime‚450]¿Äg§'Ú¡O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36783420721152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172110848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4365825 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34095984558080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36225102155776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚ 450.220664161 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 558301526016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35614778243072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8992325 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4365828 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36225102155776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16100727 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7098425 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2169433 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7098427 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34095959654400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35614761203712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚74 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7098425 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8992325 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 558301526016 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34674047574528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 556246178101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9002300 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36783315745792 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 577890905600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16090750 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7098427 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 578063016448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6535261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197014528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34673850560000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36225119195136 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 558213590016 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36783315745792 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9002300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 556332315183 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843685087762296e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36171007381813 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6535031 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2169206æ+c«'Û¥O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2169206 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36783420721152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172110848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4365825 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34095984558080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36225102155776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚ 450.220664161 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 558301526016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35614778243072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8992325 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4365828 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36225102155776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16100727 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7098425 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2169433 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7098427 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34095959654400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35614761203712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚74 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7098425 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8992325 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 558301526016 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34674047574528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 556246178101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9002300 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36783315745792 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 577890905600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16090750 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7098427 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 578063016448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6535261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197014528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34673850560000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36225119195136 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 558213590016 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36783315745792 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9002300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 556332315183 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843685087762296e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36171007381813 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6535031 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1JôWÇc0Ü+ _wandbruntime‚450  _runtime‚450³ôÿoaݪM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36262213120000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6541681 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34130896072704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34130871169024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36262230159360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16117243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556899636825 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16117243 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34709550066176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytÔzOes.large_pool.freed‚35651233239040 _step‚75 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36208046738671 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558870892544 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7105646 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9001606 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9011595 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6541922 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚36821101051904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9011595 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36820996069376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚ 450.559046881 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16107252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 556813499631 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36262213120000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7105648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203298816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36208149915225 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36821101051904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558870892544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7105648 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 578475598336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 $ _timestamp‚1.784368509114939e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4370312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35651250278400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9011595 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚36820996069376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚ 450.559046881 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16107252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 556813499631 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36262213120000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7105648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203298816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36208149915225 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36821101051904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558870892544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7105648 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 578475598336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 $ _timestamp‚1.784368509114939e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4370312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35651250278400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6541681 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34130896072704 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34130871169024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36262230159360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16117243 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556899636825 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16117243 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34709550066176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35651233239040 _step‚75 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36208046738671 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473K[JLØ0ß+ _wandbruntime‚450  _runtime‚450IŒ…àý /loss/FRt1/detection_loss‚0.6108467374156429 /loss/FBt2/detection_loss‚0.6712244760895595 trainer/global_step‚949 % _timestamp‚1.7843685091165497e+09 _step‚76 0loss/FRt2/assignment_loss‚2.0056072176506223 2loss/SRqqt2/assignment_loss‚1.1735677260923563 0loss/FBt2/assignment_loss‚0.1279138936524268 /loss/FRt2/detection_loss‚1.3295445974496027 _runtime‚ 450.560081003 2loss/SRqqt1/assignment_loss‚1.3581900718743387 1loss/FBt1/assignment_loss‚0.41499316709454986 1loss/SRqqt1/detection_loss‚1.5435805502145186 /loss/FBt1/detection_loss‚2.2412574346128262 1loss/SRqqt2/detection_loss‚0.7543436089411915 &loss/total_loss‚15.545368809908643 epoch‚1 /loss/FRt1/assignment_loss‚3.314299328821008žWá 2loss/SRqqt1/assignment_loss‚1.3581900718743387 1loss/FBt1/assignment_loss‚0.41499316709454986 1loss/SRqqt1/detection_loss‚1.5435805502145186 /loss/FBt1/detection_loss‚2.2412574346128262 1loss/SRqqt2/detection_loss‚0.7543436089411915 &loss/total_loss‚15.545368809908643 epoch‚1 /loss/FRt1/assignment_loss‚3.314299328821008 /loss/FRt1/detection_loss‚0.6108467374156429 /loss/FBt2/detection_loss‚0.6712244760895595 trainer/global_step‚949 % _timestamp‚1.7843685091165497e+09 _step‚76 0loss/FRt2/assignment_loss‚2.0056072176506223 2loss/SRqqt2/assignment_loss‚1.1735677260923563 0loss/FBt2/assignment_loss‚0.1279138936524268 /loss/FRt2/detection_loss‚1.3295445974496027 _runtime‚ 450.560081003LÊŽ-‘â‚(:† ‹›íÒóÓçŠ gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̇Á½ã‚(:² ‹›íÒîÔÛŒ proc.cpu.threads‚17( proc.memory.availableMB‚ 437.96484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚2077.57763671875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2646.90234375 cpu‚26.83724840300651 disk.sda3.out‚ 14.67578125 disk.sda.out‚ 14.67578125" disk.nvme0c0n1.in‚ 607.94921875 disk.sda3.in‚0 disk.rbd1.out‚1 disk.rbd1.in‚0$ memory_percent‚97.32687473297119 network.recv‚189611 disk.dm-1.out‚ 14.67578125 disk.dm-0.in‚ 607.94921875 disk.drbd2165.in‚0 network.sent‚ 1.042497e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚39.604187365729175$ disk./.usageGB‚590.1310119628906 disk.sda2.out‚0 disk.nvme0n1.in‚ 607.94921875% disk.nvme0n1.out‚2077.57763671875* proc.memory.percent‚16.155409812927246 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚2077.57763671875¡_ž¥0ä+ _wandbruntime‚471  _runtime‚471R0ªÄå¾ _step‚77 & lr-AdamW/pg1‚0.0008480074667151374 & lr-AdamW/pg2‚0.0008480074667151374 trainer/global_step‚999 % _timestamp‚1.7843685305034792e+09 _runtime‚ 471.947302332ø|+íÈæ & lr-AdamW/pg1‚0.0008480074667151374 & lr-AdamW/pg2‚0.0008480074667151374 trainer/global_step‚999 % _timestamp‚1.7843685305034792e+09 _runtime‚ 471.947302332 _step‚77MB0ç+ _wandbruntime‚471  _runtime‚471Ö¡ØÔ§'è¡O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9468202 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843685305046272e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38667469157376 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7459477 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6868008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚999 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38080667406336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 584730294891 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16927679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9458227 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38667469157376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37438380009472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38080667406336 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 586713815040 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6868236 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 586713815040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38667364182016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7459475 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2277983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38080650366976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 607530882560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38080650366976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4590028 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35841560289280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35841535385600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 607354577408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9458227 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4590025 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36449091171840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38023110304363 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 584644157809 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201208832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176305152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38023007127921 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _step‚78 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38667364182016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2278208 _runtime‚ 471.951621788 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4590028 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35841560289280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35841535385600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 607354577408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9458227 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4590025 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36449091171840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38023110304363 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 584644157809 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201208832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176305152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38023007127921 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _step‚78 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38667364182016 G?DeviceStatsMonitor.on_traiM–Ü$n_batch_start/inactive_split.all.peak‚677 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2278208 _runtime‚ 471.951621788 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38667364182016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7459475 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2277983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38080650366976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 607530882560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38080650366976 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16944195 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9477497 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4594512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 587283174400 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38117778370560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 585211479339 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38705149488128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38705149488128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38117761331200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9467508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4594509 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 607945923072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7466698 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9477497 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16934204 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6874585 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚ 472.293142603 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7466696 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36484392823296 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38060149661333”Cƒµ&ì¯M UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 587283174400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37474852044800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16944195 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9477497 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4594512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 587283174400 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38117778370560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 585211479339 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38705149488128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38705149488128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38117761331200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9467508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4594509 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 607945923072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7466698 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9477497 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16934204 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6874585 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚ 472.293142603 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7466696 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36484392823296 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38060149661333 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37474835005440 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128O}Ô’ 0í+ _wandbruntime‚472  _runtime‚472¹¢yîþ 0loss/FRt1/assignment_loss‚3.3489766744744807 0loss/FRt2/assignment_loss‚1.8563998362847738 1loss/FBt1/assignment_loss‚0.16179605260752591 1loss/FBt2/assignment_loss‚0.10014886110366206 /loss/FRt1/detection_loss‚0.6132061549135113 1loss/SRqqt2/detection_loss‚0.7703967139851756 /loss/FBt2/detection_loss‚0.6841081849594013 epoch‚1 1loss/SRqqt1/assignment_loss‚1.273939852901506 &loss/total_loss‚14.584642965168927 trainer/global_step‚999 2loss/SRqqt2/assignment_loss‚1.0103439668043286 /loss/FRt2/detection_loss‚1.4724550536941803 1loss/SRqqt1/detection_loss‚1.2824061468899626 /loss/FBt1/detection_loss‚2.0104654665504182 % _timestamp‚1.7843685308507607e+09 _runtime‚ 472.294368137 _step‚80d÷/(ï‚ 1loss/SRqqt1/detection_loss‚1.2824061468899626 /loss/FBt1/detection_loss‚2.0104654665504182 % _timestamp‚1.7843685308507607e+09 _runtime‚ 472.294368137 _step‚80 0loss/FRt1/assignment_loss‚3.3489766744744807 0loss/FRt2/assignment_loss‚1.8563998362847738 1loss/FBt1/assignment_loss‚0.16179605260752591 1loss/FBt2/assignment_loss‚0.10014886110366206 /loss/FRt1/detection_loss‚0.6132061549135113 1loss/SRqqt2/detection_loss‚0.7703967139851756 /loss/FBt2/detection_loss‚0.6841081849594013 epoch‚1 1loss/SRqqt1/assignment_loss‚1.273939852901506 &loss/total_loss‚14.584642965168927 trainer/global_step‚999 2loss/SRqqt2/assignment_loss‚1.0103439668043286 /loss/FRt2/detection_loss‚1.4724550536941803Ppìw‘ð‚(:† š›íÒõîñŠ gpu.0.gpu‚20.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªp¬¾ñ‚(:³ š›íÒ±ÓòŒ" disk.dm-0.out‚2081.76123046875 proc.memory.rssMB‚ 2646.90625 disk.sda.in‚0 disk.nvme0n1.in‚ 607.94921875 disk.drbd2165.in‚0# memory_percent‚97.3271131515503 disk.dm-0.in‚ 607.94921875$ disk./.usageGB‚590.1310691833496 disk.drbd2165.out‚0 disk.dm-1.out‚ 14.89453125 network.recv‚195027 disk.rbd1.out‚1.0625 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.155433654785156 network.sent‚ 1.105565e+06 disk.sda2.in‚0( proc.memory.availableMB‚ 437.92578125 cpu‚26.82036918845286 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 14.89453125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚2081.76123046875 disk.dm-1.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚39.604191205842184 disk.sda.out‚ 14.89453125" disk.nvme0c0n1.in‚ 607.94921875% disk.nvme0n1.out‚2081.76123046875 disk.rbd0.in‚0 disk.sda2.out‚0ŸÊ50ò+ _wandbruntime‚493  _runtime‚493HS®oÃó½ trainer/global_step‚1049 % _timestamp‚1.7843685521698644e+09 _runtime‚ 493.613641397 _step‚81 % lr-AdamW/pg1‚0.000847977177290714 % lr-AdamW/pg2‚0.000847977177290714In¥ÇôÁ % lr-AdamW/pg2‚0.000847977177290714 trainer/global_step‚1049 % _timestamp‚1.7843685521698644e+09 _runtime‚ 493.613641397 _step‚81 % lr-AdamW/pg1‚0.000847977177290714Qåüð0õ+ _wandbruntime‚493  _runtime‚493} !‚oö¡O Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40551412618240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39936215617536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4814225 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 493.61484032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615301976064 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615301976064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9924129 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 NFDeviceStatsMonitor.on_train_batcSÀÉ8h_start/inactive_split.large_pool.peak‚231 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 636928585216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚1049 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17754631 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40551412618240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 613042137517 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7820525 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9934104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4814228 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176305152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39261981775872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 613128274599 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7200441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9924129 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 615214040064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38223863396864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17744654 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39875110050471 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39875006874029 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37587136020480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40551517593600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201208832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39261964736512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39936215617536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7820527 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7200216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39936198578176 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 615214040064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40551517593600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2385991 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2386213 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38224064605696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37587111116800 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9934104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4814228 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176305152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39261981775872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 613128274599 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7200441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9924129 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 615214040064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38223863396864 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17744654 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39875110050471 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39875006874029 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37587136020480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40551517593600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201208832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39261964736512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39936215617536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7820527 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7200216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39936198578176 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 615214040064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40551517593600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2385991 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2386213 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38224064605696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37587111116800 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40551412618240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39936215617536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4814225 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 493.61484032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615301976064 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615301976064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9924129 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146Rÿ‹Å0ø+ _wandbruntime‚493  _runtime‚493Ú;-²&ù¬M FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7827748 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9943399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9933410 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17771147 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9933410 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40589197924352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2388331 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37622047535104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚ 493.949363618 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17771147 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39298453811200 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40589197924352 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7827746 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637338690048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613609459047 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4818709 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39973309542400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7827746 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37622022631424 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39912046230887 trainer/global_step‚1049 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38259361321472 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17761156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚83 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7827748 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2388104 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615783399424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39298436771840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 615871342592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7206813 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178395136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843685525053208e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39973326581760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192,Qšú°M JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚83 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7827748 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2388104 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615783399424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39298436771840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 615871342592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7206813 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178395136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843685525053208e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39973326581760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7827748 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9943399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9933410 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17771147 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9933410 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40589197924352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2388331 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37622047535104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚ 493.949363618 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17771147 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39298453811200 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40589197924352 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7827746 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637338690048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613609459047 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4818709 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39973309542400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7827746 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37622022631424 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39912046230887 trainer/global_step‚1049 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38259361321472 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17761156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936SP–N¡0û+ _wandbruntime‚493  _runtime‚493Ÿ"9üý trainer/global_step‚1049 % _timestamp‚1.7843685525072494e+09 1loss/FBt1/assignment_loss‚0.18801219334515432 /loss/FRt1/detection_loss‚0.6300014294096751 &loss/total_loss‚14.456710456261343 _runtime‚ 493.950794178 /loss/FRt1/assignment_loss‚3.188821946581205 /loss/FRt2/assignment_loss‚1.814911834553358 2loss/SRqqt1/assignment_loss‚1.2560736364615497 0loss/FBt2/assignment_loss‚0.0368565587035418 /loss/FRt2/detection_loss‚1.4628588819428752 1loss/SRqqt1/detection_loss‚1.3400395930146993 1loss/SRqqt2/detection_loss‚0.7609527608962421 /loss/FBt1/detection_loss‚1.9376393039193418 _step‚84 2loss/SRqqt2/assignment_loss‚0.9234041389355555 /loss/FBt2/detection_loss‚0.9171381784981445 epoch‚1 ƒKý /loss/FRt2/assignment_loss‚1.814911834553358 2loss/SRqqt1/assignment_loss‚1.2560736364615497 0loss/FBt2/assignment_loss‚0.0368565587035418 /loss/FRt2/detection_loss‚1.4628588819428752 1loss/SRqqt1/detection_loss‚1.3400395930146993 1loss/SRqqt2/detection_loss‚0.7609527608962421 /loss/FBt1/detection_loss‚1.9376393039193418 _step‚84 2loss/SRqqt2/assignment_loss‚0.9234041389355555 /loss/FBt2/detection_loss‚0.9171381784981445 epoch‚1 trainer/global_step‚1049 % _timestamp‚1.7843685525072494e+09 1loss/FBt1/assignment_loss‚0.18801219334515432 /loss/FRt1/detection_loss‚0.6300014294096751 &loss/total_loss‚14.456710456261343 _runtime‚ 493.950794178 /loss/FRt1/assignment_loss‚3.188821946581205T¨þC€‘þ‚(:† ©›íÒºÉòŒ gpu.0.gpu‚25.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d°a„²ÿ‚(:§ ©›íҨ͎ network.recv‚200598 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 15.078125* proc.memory.percent‚16.155433654785156 cpu‚26.804297820801985$ disk./.usageGB‚590.1309623718262 disk.sda.in‚0 proc.memory.rssMB‚ 2646.90625 proc.cpu.threads‚17 disk.drbd2165.in‚0) disk./.usagePercent‚39.60418403763122 disk.rbd1.out‚1.0625" disk.nvme0c0n1.in‚ 607.94921875 disk.dm-1.in‚0 disk.dm-0.out‚2085.970703125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚2085.970703125 disk.nvme0n1.in‚ 607.94921875# memory_percent‚97.3257303237915 disk.sda2.in‚0 disk.sda3.out‚ 15.078125 disk.sda.out‚ 15.078125 disk.rbd1.in‚0( proc.memory.availableMB‚ 438.15234375 disk.sda1.out‚0 disk.dm-0.in‚ 607.94921875 disk.sda3.in‚0# disk.nvme0n1.out‚2085.970703125 network.sent‚ 1.181335e+06q¬-€‚(:… ¸›íÒÀØŠ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íþjG¾‚(:³ ¸›íÒÊÖ disk.dm-0.in‚ 671.90234375 disk.nvme0n1.in‚ 671.90234375 disk.sda3.out‚ 15.43359375 disk.drbd2165.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚2093.48095703125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.in‚0$ memory_percent‚97.32882976531982 cpu‚26.789446284354636 disk.sda1.out‚0 disk.sda.out‚ 15.43359375% disk.nvme0n1.out‚2093.48095703125( proc.memory.availableMB‚ 437.64453125 network.sent‚ 1.233192e+06 disk.sda2.out‚0" disk.dm-0.out‚2093.48095703125 disk.drbd2165.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2646.90625* proc.memory.percent‚16.155433654785156) disk./.usagePercent‚39.60417123725451 disk.dm-1.in‚0 disk.dm-1.out‚ 15.43359375 disk.rbd1.out‚1.125" disk.nvme0c0n1.in‚ 671.90234375 proc.cpu.threads‚17 network.recv‚206213$ disk./.usageGB‚590.1307716369629œ¥l²0‚+ _wandbruntime‚514  _runtime‚514S—”­Äƒ¾ & lr-AdamW/pg2‚0.0008479400974974915 trainer/global_step‚1099 $ _timestamp‚1.784368573556105e+09 _runtime‚ 514.999936661 _step‚85 & lr-AdamW/pg1‚0.0008479400974974915fâ6hÈ„Â & lr-AdamW/pg2‚0.0008479400974974915 trainer/global_step‚1099 $ _timestamp‚1.784368573556105e+09 _runtime‚ 514.999936661 _step‚85 & lr-AdamW/pg1‚0.0008479400974974915U(#—×0…+ _wandbruntime‚515  _runtime‚515pÑ)«'†¥O L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7533107 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 643714265088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784368573557237e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39999105436160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8181577 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8181575 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 641526254307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 643714265088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39332686848000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 643802201088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 666219476480 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 641440117225 _runtime‚ 515.001215065 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39332711751680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41727006620137 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42435461054464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5038425 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7533332 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2494682 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41085566502912 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18571606 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41791746789376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8181575 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42435461054464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 643802201088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41727109796579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10400006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41791763828736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 666393684480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42435461054464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5038425 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7533332 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2494682 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41085566502912 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18571606 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41791746789376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8181575 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42435461054464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 643802201088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41727109796579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10400006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41791763828736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 666393684480 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 643714265088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784368573557237e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39999105436160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8181577 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8181575 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 641526254307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 643714265088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39332686848000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 643802201088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 666219476480 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 641440117225 _runtime‚ 515.001215065 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0V2TQ=0ˆ+ _wandbruntime‚515  _runtime‚515³5Gò±‰°M SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42473246360576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18588108 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41828874792960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8188798 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7539962 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 666985477632 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8188798 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39367598362624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41764045976995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 644371567616 UHDeviceStatsMonitor.ð Ò¢on_train_batch_end/allocated_bytes.small_pool.current‚87943168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8188796 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41828857753600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10399312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18588108 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42473246360576 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 644283624448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 515.336609203 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18598099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41122055577600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41828874792960 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 644283624448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41764149153549 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 642093575949 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8188796 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 % _timestamp‚1.7843685738925028e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39367623266304 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7539725 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2497050 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666809179648 _step‚87 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 642007438755 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5042912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18598099 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41122038538240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10409301 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40034608743936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42473141378048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5042909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚1099 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 644371567616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8188796 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41828857753600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10399312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18588108 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42473246360576 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 644283624448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 515.336609203 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18598099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41122055577600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41828874792960 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 644283624448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41764149153549 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 642093575949 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8188796 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 % _timestamp‚1.7843685738925028e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39367623266304 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7539725 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2497050 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666809179648 _step‚87 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 642007438755 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5042912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18598099 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41122038538240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10409301 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40034608743936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42473141378048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5042909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚1099 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 644371567616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42473246360576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18588108 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41828874792960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8188798 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0WIÚ†0‹+ _wandbruntime‚515  _runtime‚515=Ø®Œý 0loss/FBt1/assignment_loss‚0.3153895021006021 /loss/FRt1/detection_loss‚0.5909513336826654 .loss/FBt1/detection_loss‚2.079120058388937 _step‚88 /loss/FRt1/assignment_loss‚3.221089421369352 0loss/FRt2/assignment_loss‚1.7847334725696713 2loss/SRqqt1/assignment_loss‚1.1923650257041056 /loss/FRt2/detection_loss‚1.3991620969701914 1loss/SRqqt2/detection_loss‚0.7069362875185805 /loss/FBt2/detection_loss‚0.8498651215511306 1loss/SRqqt1/detection_loss‚1.1617657712101936 epoch‚1 trainer/global_step‚1099 % _timestamp‚1.7843685738939724e+09 _runtime‚ 515.337561341 2loss/SRqqt2/assignment_loss‚0.7742055494418211 1loss/FBt2/assignment_loss‚0.09837465142328752 &loss/total_loss‚14.173958291930537ÿE`m trainer/global_step‚1099 % _timestamp‚1.7843685738939724e+09 _runtime‚ 515.337561341 2loss/SRqqt2/assignment_loss‚0.7742055494418211 1loss/FBt2/assignment_loss‚0.09837465142328752 &loss/total_loss‚14.173958291930537 0loss/FBt1/assignment_loss‚0.3153895021006021 /loss/FRt1/detection_loss‚0.5909513336826654 .loss/FBt1/detection_loss‚2.079120058388937 _step‚88 /loss/FRt1/assignment_loss‚3.221089421369352 0loss/FRt2/assignment_loss‚1.7847334725696713 2loss/SRqqt1/assignment_loss‚1.1923650257041056 /loss/FRt2/detection_loss‚1.3991620969701914 1loss/SRqqt2/detection_loss‚0.7069362875185805 /loss/FBt2/detection_loss‚0.8498651215511306 1loss/SRqqt1/detection_loss‚1.1617657712101936 epoch‚1X«¦“ä‘Ž‚(:† Ç›íÒÿÈé‹ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚162.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü?…cº‚(:¯ Ç›íÒ˜¯ proc.cpu.threads‚17 network.recv‚210434 disk.sda3.in‚0 disk.sda.out‚ 15.7109375( proc.memory.availableMB‚ 437.50390625 disk.rbd1.out‚1.125 network.sent‚ 1.236708e+06 disk.sda3.out‚ 15.7109375 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 671.90234375 disk.dm-1.in‚0 disk.dm-0.in‚ 671.90234375$ memory_percent‚97.32968807220459 proc.memory.rssMB‚ 2646.90625" disk.dm-0.out‚2097.62744140625# disk./.usageGB‚590.130542755127 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚26.774785039522417' disk.nvme0c0n1.out‚2097.62744140625% disk.nvme0n1.out‚2097.62744140625 disk.dm-1.out‚ 15.7109375 disk.nvme0n1.in‚ 671.90234375 disk.rbd0.out‚0 disk.sda2.out‚0) disk./.usagePercent‚39.60415587680245* proc.memory.percent‚16.155433654785156 disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0^Ùƒ0+ _wandbruntime‚537  _runtime‚537/z†sÄ‘¾ & lr-AdamW/pg2‚0.0008478962279293781 trainer/global_step‚1149 $ _timestamp‚1.784368595729267e+09 _runtime‚ 537.173096324 _step‚89 & lr-AdamW/pg1‚0.0008478962279293781#Š[úÈ’Â trainer/global_step‚1149 $ _timestamp‚1.784368595729267e+09 _runtime‚ 537.173096324 _step‚89 & lr-AdamW/pg1‚0.0008478962279293781 & lr-AdamW/pg2‚0.0008478962279293781YñÄ—80“+ _wandbruntime‚537  _runtime‚537µˆ|¬'”¦O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43647295000576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 669924234015 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 672214490112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10855933 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚1149 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43647312039936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5262625 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44319509490688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192820224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7865993 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚90 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42909168269312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10865908 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43579006366245 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43579109542687 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44319614466048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 695871671296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 672302426112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5262628 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 672302426112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2603598 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 695703754752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8542625 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43647312039936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43647295000576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8542625 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.allªùêFç!.current‚233 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 669924234015 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 672214490112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10855933 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚1149 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43647312039936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5262625 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44319509490688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192820224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7865993 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚90 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42909168269312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10865908 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43579006366245 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43579109542687 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44319614466048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 695871671296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 672302426112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5262628 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 672302426112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2603598 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 695703754752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8542625 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43647312039936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43647295000576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8542625 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43684405964800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44357294796800 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 672783849472 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 696461469696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8549848 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8549846 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10865214 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843685960749462e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 672871792640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19425051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190715904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41113198997504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165812224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19425051 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7872670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19415060 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8549848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚91 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 672783849472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 696295657472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10865214 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5267109 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43616045723103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2605802 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41809660467200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8549846 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42945657344000 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 670491555657*³5º&˜´M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19415060 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2605561 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42945640304640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43684423004160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10875203 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41809469751296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 672783849472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 696295657472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10865214 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5267109 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43616045723103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2605802 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41809660467200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8549846 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42945657344000 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 670491555657 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10875203 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41113174093824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43684405964800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44357189814272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43684405964800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44357294796800 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 672783849472 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 696461469696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8549848 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8549846 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10865214 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843685960749462e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 672871792640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19425051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190715904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41113198997504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165812224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19425051 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7872670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19415060 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8549848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚91 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 EEú0£+ _wandbruntime‚559  _runtime‚559¿Û€¬'¤¦O YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 700714715136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165819392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 698322213723 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 725319309824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46203557926912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8903677 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45431109288795 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8903675 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45502843211776 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 700802651136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20235487 _step‚94 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44732770035712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11331810 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46203662902272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8903675 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42823863214080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 700802651136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46203557926912 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5486828 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43548991800832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45502860251136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5486825 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2711784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8198612 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44732787075072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45502843211776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2711576 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190723072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 559.524907696 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20225510 % _timestamp‚1.7843686180807872e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11331810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42823838310400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 725153490432 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46203662902272 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 700714715136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11321835 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚1199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45431006112353 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8198401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8903677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45502860251136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚211 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11321835 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0˜ŸÚé°'¥ªO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44732770035712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11331810 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46203662902272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8903675 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42823863214080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 700802651136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46203557926912 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5486828 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43548991800832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45502860251136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5486825 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2711784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8198612 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44732787075072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45502843211776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2711576 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190723072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 559.524907696 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20225510 % _timestamp‚1.7843686180807872e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11331810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42823838310400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 725153490432 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46203662902272 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 700714715136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11321835 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚1199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45431006112353 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8198401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8903677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45502860251136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚211 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11321835 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 700714715136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165819392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 698322213723 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 725319309824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46203557926912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8903677 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45431109288795 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8903675 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45502843211776 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 700802651136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20235487 _step‚94^¼©í0¦+ _wandbruntime‚559  _runtime‚559öÜyµ&§¯M NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8205214 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 698889535365 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43584689288704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42858774728704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5491312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45468045469211 trainer/global_step‚1199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2713672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46241343233024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20252003 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8910898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20242012 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8910896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11331116 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2713902 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 698803398171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11341105 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 725914560000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45539971215360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45539971215360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43584488087040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚95¿À°uÚ¨³M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚95 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8205214 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 698889535365 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43584689288704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42858774728704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_endÏ ÇÂß/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5491312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45468045469211 trainer/global_step‚1199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2713672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46241343233024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20252003 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8910898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20242012 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8910896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11331116 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2713902 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 698803398171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11341105 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 725914560000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45539971215360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45539971215360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43584488087040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0_lÕ0©+ _wandbruntime‚559  _runtime‚5599éyªý /loss/FRt1/assignment_loss‚3.166045488870662 1loss/FBt1/assignment_loss‚0.05845342232813974 1loss/SRqqt2/detection_loss‚0.7447935276086555 trainer/global_step‚1199 _runtime‚ 559.887550827 &loss/total_loss‚13.766387137107234 0loss/FRt2/assignment_loss‚1.7416322383474796 1loss/SRqqt1/assignment_loss‚1.306375579403366 /loss/FRt1/detection_loss‚0.6536276273504051 /loss/FRt2/detection_loss‚1.3874202011308323 1loss/SRqqt1/detection_loss‚1.4164638307766388 /loss/FBt2/detection_loss‚0.8635419246798847 2loss/SRqqt2/assignment_loss‚0.7990138003331415 epoch‚1 % _timestamp‚1.7843686184439511e+09 1loss/FBt2/assignment_loss‚0.14820989847153215 .loss/FBt1/detection_loss‚1.480809597806497 _step‚96ÿñ« 1loss/FBt2/assignment_loss‚0.14820989847153215 .loss/FBt1/detection_loss‚1.480809597806497 _step‚96 /loss/FRt1/assignment_loss‚3.166045488870662 1loss/FBt1/assignment_loss‚0.05845342232813974 1loss/SRqqt2/detection_loss‚0.7447935276086555 trainer/global_step‚1199 _runtime‚ 559.887550827 &loss/total_loss‚13.766387137107234 0loss/FRt2/assignment_loss‚1.7416322383474796 1loss/SRqqt1/assignment_loss‚1.306375579403366 /loss/FRt1/detection_loss‚0.6536276273504051 /loss/FRt2/detection_loss‚1.3874202011308323 1loss/SRqqt1/detection_loss‚1.4164638307766388 /loss/FBt2/detection_loss‚0.8635419246798847 2loss/SRqqt2/assignment_loss‚0.7990138003331415 epoch‚1 % _timestamp‚1.7843686184439511e+09`\Bx,¬‚(:„ ô›íÒ•óÞ‹ gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚178.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ù³­‚(:¨ ô›íÒ³Ô  disk.drbd2165.out‚0& disk.nvme0c0n1.out‚2110.2685546875 disk.nvme0n1.in‚ 750.421875 disk.sda1.in‚0 disk.dm-1.out‚ 16.69140625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚97.33285903930664$ disk./.usageGB‚590.1305541992188 disk.sda2.out‚0 disk.dm-0.in‚ 750.421875 disk.nvme0c0n1.in‚ 750.421875 disk.rbd1.out‚1.375 disk.rbd0.out‚0* proc.memory.percent‚16.155433654785156 cpu‚26.734029494279792 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2646.90625 network.sent‚ 1.367815e+06) disk./.usagePercent‚39.60415664482506! disk.dm-0.out‚2110.2685546875 disk.sda.in‚0$ disk.nvme0n1.out‚2110.2685546875 disk.sda2.in‚0 disk.sda3.out‚ 16.69140625 disk.sda.out‚ 16.69140625& proc.memory.availableMB‚ 436.984375 network.recv‚224906A¤0®+ _wandbruntime‚581  _runtime‚581n»ëÁ¯» _runtime‚ 581.860623147 _step‚97 $ lr-AdamW/pg1‚0.00084778812238786 $ lr-AdamW/pg2‚0.00084778812238786 trainer/global_step‚1249 % _timestamp‚1.7843686404159186e+09”¢Å°¿ _step‚97 $ lr-AdamW/pg1‚0.00084778812238786 $ lr-AdamW/pg2‚0.00084778812238786 trainer/global_step‚1249 % _timestamp‚1.7843686404159186e+09 _runtime‚ 581.860623147a„l 0±+ _wandbruntime‚581  _runtime‚581ßÂfr¬'²¦O @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 % _timestamp‚1.7843686404166474e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9264727 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21052462 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9264725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11797712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11787737 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 729214940160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 754627856384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2820711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45324241009664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48087606363136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21062439 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8531522 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46556388841472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48087711338496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47358391422976 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729302876160 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48087606363136 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726634056349 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47358408462336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚1249 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 729214940160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47358391422976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5711025 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 754802064384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135ÿ&ð5°'³ªO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11797712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46556371802112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11787737 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2820711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45324241009664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48087606363136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21062439 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8531522 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46556388841472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48087711338496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47358391422976 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729302876160 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48087606363136 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726634056349 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47358408462336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚1249 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 729214940160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47358391422976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5711025 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 754802064384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 % _timestamp‚1.7843686404166474e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9264727 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21052462 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9264725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11797712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11787737 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 729214940160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 754627856384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8538140 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729872242688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197007360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9271948 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11807007 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 729872242688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46592860876800 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚1249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47395519426560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47320045215319 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21078955 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5715512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2822631 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11797018 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45359739459584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45359542452224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44604350459904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44604325556224 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9271946 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11797018 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48125391669248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9271948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8538356 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5715509 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 755216896000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 755388999680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21068964 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48125286686720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21078955 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47320148391873 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 727287515073 % _timestamp‚1.7843686407570274e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 582.201119899 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47395502387200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2822844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚99 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47395519426560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11797018 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45359739459584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45359542452224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44604350459904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44604325556224 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9271946 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11797018 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48125391669248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9271948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8538356 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5715509 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 755216896000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 755388999680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21068964 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48125286686720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21078955 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47320148391873 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 727287515073 % _timestamp‚1.7843686407570274e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 582.201119899 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47395502387200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2822844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚99 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47395519426560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8538140 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729872242688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197007360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9271948 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11807007 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 729872242688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46592860876800 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚1249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47395519426560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231cˆ†/›0·+ _wandbruntime‚582  _runtime‚582Á{k¸€ /loss/FRt1/assignment_loss‚3.037953648765014 /loss/FRt2/detection_loss‚1.4153724481412193 trainer/global_step‚1249 0loss/FRt2/assignment_loss‚1.7335160456409524 1loss/FBt1/assignment_loss‚0.09734304852666836 /loss/FRt1/detection_loss‚0.6641026310323774 /loss/FBt2/detection_loss‚0.7682619578737233 &loss/total_loss‚13.179413456114693 % _timestamp‚1.7843686407587366e+09 _runtime‚ 582.202273446 2loss/SRqqt2/assignment_loss‚0.7415366696092762 1loss/FBt2/assignment_loss‚0.09877975239099775 1loss/SRqqt1/detection_loss‚1.4533162153977202 1loss/SRqqt2/detection_loss‚0.8189290709454905 /loss/FBt1/detection_loss‚1.3057423961513184 epoch‚1 _step‚100 2loss/SRqqt1/assignment_loss‚1.0445595716399358 - ¹„ epoch‚1 _step‚100 2loss/SRqqt1/assignment_loss‚1.0445595716399358 /loss/FRt1/assignment_loss‚3.037953648765014 /loss/FRt2/detection_loss‚1.4153724481412193 trainer/global_step‚1249 0loss/FRt2/assignment_loss‚1.7335160456409524 1loss/FBt1/assignment_loss‚0.09734304852666836 /loss/FRt1/detection_loss‚0.6641026310323774 /loss/FBt2/detection_loss‚0.7682619578737233 &loss/total_loss‚13.179413456114693 % _timestamp‚1.7843686407587366e+09 _runtime‚ 582.202273446 2loss/SRqqt2/assignment_loss‚0.7415366696092762 1loss/FBt2/assignment_loss‚0.09877975239099775 1loss/SRqqt1/detection_loss‚1.4533162153977202 1loss/SRqqt2/detection_loss‚0.8189290709454905 /loss/FBt1/detection_loss‚1.3057423961513184d”›4I‘º‚(:† ƒœíÒ„ö«‹ gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\r¤Â·»‚(:¬ ƒœíÒز¹ network.sent‚ 1.431607e+06 disk.sda.in‚0 disk.sda.out‚ 17.08203125$ disk.nvme0n1.out‚2113.6357421875$ memory_percent‚97.33145236968994# proc.memory.rssMB‚ 2646.91015625 disk.sda3.out‚ 17.08203125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 750.421875 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.60405270576616 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 17.08203125$ disk./.usageGB‚590.1290054321289 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 750.421875* proc.memory.percent‚16.155457496643066 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 437.21484375 disk.sda2.out‚0 disk.dm-0.in‚ 750.421875 disk.rbd1.out‚1.375 disk.rbd0.in‚0 disk.sda2.in‚0! disk.dm-0.out‚2113.6357421875 network.recv‚232044& disk.nvme0c0n1.out‚2113.6357421875 cpu‚26.721344896780886ÂΑ¼‚(:† ’œíÒû™¬Š gpu.0.gpu‚64.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚171.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0] ù³½‚(:¨ ’œíÒä‹òŒ( proc.memory.availableMB‚ 437.38671875 network.recv‚237203 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 750.421875 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 17.28515625# proc.memory.rssMB‚ 2646.91015625* disk./.usagePercent‚39.604084450700405 disk.rbd0.out‚0# disk.nvme0n1.out‚2118.376953125$ disk./.usageGB‚590.1294784545898 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 750.421875# memory_percent‚97.3304033279419 network.sent‚ 1.494237e+06 disk.sda1.out‚0 disk.dm-0.in‚ 750.421875 disk.sda3.out‚ 17.28515625% disk.nvme0c0n1.out‚2118.376953125 cpu‚26.710130339655827 disk.dm-0.out‚2118.376953125 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 17.28515625 disk.rbd1.out‚1.5* proc.memory.percent‚16.155457496643066 proc.cpu.threads‚17a™W0¾+ _wandbruntime‚604  _runtime‚604òÁ\’Å¿¿ & lr-AdamW/pg1‚0.0008477238881459849 & lr-AdamW/pg2‚0.0008477238881459849 trainer/global_step‚1299 % _timestamp‚1.7843686629219577e+09 _runtime‚ 604.36579713 _step‚101¥¾j¹ÉÀà _step‚101 & lr-AdamW/pg1‚0.0008477238881459849 & lr-AdamW/pg2‚0.0008477238881459849 trainer/global_step‚1299 % _timestamp‚1.7843686629219577e+09 _runtime‚ 604.36579713eÚ*Uˆ0Á+ _wandbruntime‚604  _runtime‚604dòÐE"§O E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46315014676480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9625777 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49971654799360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49971759774720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21889391 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2928198 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9625775 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9625775 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49213956673536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49213939634176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12253639 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49213956673536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49213939634176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8863426 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47099338311168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 784145232384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 757803101184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46314989772800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 % _timestamp‚1.7843686629229565e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5935225 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12263614 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8863194 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5935228 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚102 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9625777 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12253639 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49971654799360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 757803101184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178402304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48379973568512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 784323634688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48379990607872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 755118173139 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDJôÊcheviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49135005604569 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 755032036057 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2927969 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 604.368016939 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47099135005184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49971759774720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ƒ‘ž©±'ëO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9625775 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9625775 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49213956673536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49213939634176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12253639 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49213956673536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49213939634176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8863426 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47099338311168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 784145232384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 757803101184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46314989772800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 % _timestamp‚1.7843686629229565e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5935225 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12263614 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8863194 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5935228 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚102 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9625777 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12253639 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49971654799360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 757803101184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178402304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48379973568512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 784323634688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48379990607872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 755118173139 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49135005604569 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 755032036057 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2927969 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 604.368016939 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47099135005184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49971759774720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46315014676480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9625777 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49971654799360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49971759774720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21889391 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2928198fj ¹Ÿ0Ä+ _wandbruntime‚604  _runtime‚604‹·L÷µ&ůM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49251067637760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21895916 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12262920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50009440105472 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8869868 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49251050598400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 784726923776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 _runtime‚ 604.712133699 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12272909 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47134837801472 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50009335122944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46349926191104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184686592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 758372467712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9632998 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48416462643200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21905907 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9632996 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21905907 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 758284524544 trainer/global_step‚1299 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 758284524544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49251050598400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2930385 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚103 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50009440105472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9632996 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49172148137981 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2930159 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47134628211200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152zò¹&ƳM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46349926191104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184686592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 758372467712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9632998 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48416462643200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21905907 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9632996 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21905907 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 758284524544 trainer/global_step‚1299 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 758284524544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49251050598400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2930385 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚103 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50009440105472 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9632996 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49172148137981 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2930159 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47134628211200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49251067637760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21895916 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12262920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50009440105472 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8869868 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49251050598400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 784726923776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 _runtime‚ 604.712133699 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12272909 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47134837801472 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50009335122944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0gÅ2$0Ç+ _wandbruntime‚604  _runtime‚604zC:pÈÿ _step‚104 1loss/FBt1/assignment_loss‚0.14022838049731945 1loss/SRqqt1/detection_loss‚1.2440850724526447 epoch‚1 0loss/FRt2/assignment_loss‚1.6194109996788115 2loss/SRqqt1/assignment_loss‚1.0599977382653052 1loss/FBt2/assignment_loss‚0.16705222028121866 1loss/SRqqt2/detection_loss‚0.7499162275605741 /loss/FRt1/detection_loss‚0.6169165248704439 /loss/FBt1/detection_loss‚1.3749285720288753 .loss/FBt2/detection_loss‚0.702350881842098 trainer/global_step‚1299 % _timestamp‚1.7843686632696922e+09 _runtime‚ 604.713223946 0loss/FRt1/assignment_loss‚3.1287738929897486 2loss/SRqqt2/assignment_loss‚0.6854605322391099 /loss/FRt2/detection_loss‚1.4685595676048913 %loss/total_loss‚12.95768061031104²müÞÜɃ /loss/FRt1/detection_loss‚0.6169165248704439 /loss/FBt1/detection_loss‚1.3749285720288753 .loss/FBt2/detection_loss‚0.702350881842098 trainer/global_step‚1299 % _timestamp‚1.7843686632696922e+09 _runtime‚ 604.713223946 0loss/FRt1/assignment_loss‚3.1287738929897486 2loss/SRqqt2/assignment_loss‚0.6854605322391099 /loss/FRt2/detection_loss‚1.4685595676048913 %loss/total_loss‚12.95768061031104 _step‚104 1loss/FBt1/assignme˜iÐÃ-nt_loss‚0.14022838049731945 1loss/SRqqt1/detection_loss‚1.2440850724526447 epoch‚1 0loss/FRt2/assignment_loss‚1.6194109996788115 2loss/SRqqt1/assignment_loss‚1.0599977382653052 1loss/FBt2/assignment_loss‚0.16705222028121866 1loss/SRqqt2/detection_loss‚0.7499162275605741hB4¡`‘Ê‚(:† ¡œíÒä¿Ú‹ gpu.0.gpu‚55.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚172.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò\׵˂(:ª ¡œíÒÛ½«# disk./.usageGB‚590.130687713623 disk.rbd1.in‚0# proc.memory.rssMB‚ 2646.91015625 cpu‚26.698963367797994 proc.cpu.threads‚17) disk./.usagePercent‚39.60416560508876 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚1.5 disk.nvme0c0n1.in‚ 750.421875 disk.dm-1.in‚0 disk.sda3.out‚ 17.51953125% disk.nvme0n1.out‚2121.64697265625 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 17.51953125 disk.sda.out‚ 17.51953125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚2121.64697265625* proc.memory.percent‚16.155457496643066 network.recv‚241638 disk.dm-0.in‚ 750.421875% proc.memory.availableMB‚ 437.28125 disk.rbd0.out‚0" disk.dm-0.out‚2121.64697265625 disk.nvme0n1.in‚ 750.421875$ memory_percent‚97.33104705810547 network.sent‚ 1.499742e+06 disk.sda3.in‚0.Ñ0Ì+ _wandbruntime‚626  _runtime‚626òArÿÆÍÀ % _timestamp‚1.7843686853233273e+09 _runtime‚ 626.767149311 _step‚105 & lr-AdamW/pg1‚0.0008476528675922507 & lr-AdamW/pg2‚0.0008476528675922507 trainer/global_step‚1349YˆÊÎÄ _runtime‚ 626.767149311 _step‚105 & lr-AdamW/pg1‚0.0008476528675922507 & lr-AdamW/pg2‚0.0008476528675922507 trainer/global_step‚1349 % _timestamp‚1.7843686853233273e+09i¼3Z»0Ï+ _wandbruntime‚626  _runtime‚626t~…­'ЧO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48060565504000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51069487845376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12719541 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51069504884736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6159428 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 786215390208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51069487845376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 % _timestamp‚1.7843686853245418e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9986827 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9195948 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3036765 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 813771159040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51855703235584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813594853888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50203575334912›ù±'Ñ«O `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48060565504000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51069487845376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12719541 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51069504884736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6159428 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 786215390208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51069487845376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 % _timestamp‚1.7843686853245418e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9986827 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9195948 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3036765 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 813771159040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51855703235584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813594853888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50203575334912j}]\Ð0Ò+ _wandbruntime‚627  _runtime‚627µ·&Ó±M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48095501922304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚107 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3039029 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184686592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51893383559168 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783997337295 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 627.122928373 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22722868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51106598809600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22722868 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843686856788332e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48909865652736 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814179043840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 trainer/global_step‚1349 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51893488541696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6163909 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9202698 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3038789 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12738811 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786872692736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9994048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22732859 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22732859 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50240047370240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51024044707535 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51106615848960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51893488541696 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48909656062464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50240064409600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12738811 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9202941 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209590272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9994046 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6163912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12728822 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51106615848960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51106598809600 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786784749568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786784749568 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786872692736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51024147884089 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784083474489 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ÞŽÔµM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843686856788332e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_‡«‹·%pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48909865652736 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814179043840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 trainer/global_step‚1349 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51893488541696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6163909 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9202698 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3038789 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12738811 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786872692736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9994048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22732859 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22732859 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50240047370240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51024044707535 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51106615848960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51893488541696 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48909656062464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50240064409600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12738811 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9202941 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209590272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9994046 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6163912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12728822 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51106615848960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51106598809600 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786784749568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786784749568 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786872692736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51024147884089 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784083474489 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48095501922304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚107 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3039029 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184686592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51893383559168 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783997337295 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 627.122928373 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22722868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51106598809600 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22722868 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040kkB0Õ+ _wandbruntime‚627  _runtime‚627Ì&ëÖÿ trainer/global_step‚1349 % _timestamp‚1.7843686856808457e+09 _step‚108 2loss/SRqqt2/assignment_loss‚0.7581355345472387 _runtime‚ 627.124374644 .loss/FRt2/detection_loss‚1.483984957528449 1loss/SRqqt1/detection_loss‚1.2084326461662311 /loss/FBt1/detection_loss‚1.3340259604156017 epoch‚1 &loss/total_loss‚13.079177177693186 0loss/FRt2/assignment_loss‚1.5487097615739738 2loss/SRqqt1/assignment_loss‚1.0508514444683552 0loss/FBt1/assignment_loss‚0.3194281808876743 1loss/FBt2/assignment_loss‚0.07256219119895349 /loss/FRt1/detection_loss‚0.5852760989369163 1loss/SRqqt2/detection_loss‚0.8360446224903721 /loss/FBt2/detection_loss‚0.7329550913176858 0loss/FRt1/assignment_loss‚3.1487706881617346Ô}d ׃ .loss/FRt2/detection_loss‚1.483984957528449 1loss/SRqqt1/detection_loss‚1.2084326461662311 /loss/FBt1/detection_loss‚1.3340259604156017 epoch‚1 &loss/total_loss‚13.079177177693186 0loss/FRt2/assignment_loss‚1.5487097615739738 2loss/SRqqt1/assignment_loss‚1.0508514444683552 0loss/FBt1/assignment_loss‚0.3194281808876743 1loss/FBt2/assignment_loss‚0.07256219119895349 /loss/FRt1/detection_loss‚0.5852760989369163 1loss/SRqqt2/detection_loss‚0.8360446224903721 /loss/FBt2/detection_loss‚0.7329550913176858 0loss/FRt1/assignment_loss‚3.1487706881617346 trainer/global_step‚1349 % _timestamp‚1.7843686856808457e+09 _step‚108 2loss/SRqqt2/assignment_loss‚0.7581355345472387 _runtime‚ 627.124374644l*øH‘Ø‚(:† °œíÒ›Éõ gpu.0.gpu‚59.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g} ¾³Ù‚(:¨ °œíÒ—ÆÒ disk.sda1.out‚0 disk.dm-0.in‚ 804.97265625! disk.dm-0.out‚2126.2158203125 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 436.64453125 network.sent‚ 1.575247e+06) disk./.usagePercent‚39.60418787774424 disk.sda.out‚17.8125 disk.drbd2165.out‚0 cpu‚26.688528537309868 network.recv‚246475 disk.nvme0n1.in‚ 804.97265625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚17.8125 disk.dm-1.out‚17.8125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 804.97265625& disk.nvme0c0n1.out‚2126.2158203125* proc.memory.percent‚16.155457496643066 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2646.91015625$ memory_percent‚97.33493328094482$ disk./.usageGB‚590.1310195922852 disk.rbd1.in‚0 disk.rbd1.out‚1.5625 proc.cpu.threads‚17$ disk.nvme0n1.out‚2126.2158203125 disk.sda1.in‚0;6~uœÚ‚(:‘ ¿œíÒéøአgpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚184.761' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.952200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wpO¼Û‚(:± ¿œíÒì·Ž disk.sda1.out‚0 network.recv‚252660$ disk./.usageGB‚590.1312294006348 disk.sda3.out‚ 18.0078125' disk.nvme0c0n1.out‚2129.35986328125 disk.rbd0.out‚0) disk./.usagePercent‚39.60420195815862 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2646.91015625 disk.sda.out‚ 18.0078125 disk.drbd2165.in‚0$ memory_percent‚97.33448028564453 disk.dm-1.in‚0% proc.memory.availableMB‚ 436.71875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd1.out‚1.5625 disk.nvme0n1.in‚ 817.62890625 network.sent‚ 1.625268e+06 disk.dm-0.in‚ 817.62890625* proc.memory.percent‚16.155457496643066 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 18.0078125" disk.dm-0.out‚2129.35986328125" disk.nvme0c0n1.in‚ 817.62890625% disk.nvme0n1.out‚2129.35986328125 cpu‚26.678215500328612 6­ 0Ü+ _wandbruntime‚649  _runtime‚6497e‹ÆÝÀ & lr-AdamW/pg1‚0.0008475750618641952 & lr-AdamW/pg2‚0.0008475750618641952 trainer/global_step‚1399 % _timestamp‚1.7843687077661853e+09 _runtime‚ 649.210031857 _step‚109ªÀÌIÊÞÄ % _timestamp‚1.7843687077661853e+09 _runtime‚ 649.210031857 _step‚109 & lr-AdamW/pg1‚0.0008475750618641952 & lr-AdamW/pg2‚0.0008475750618641952 trainer/global_step‚1399m¥+&›0ß+ _wandbruntime‚649  _runtime‚649꾋?±'à«O KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50649156152832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52027177101312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53739751671808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 814715615232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23543295 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13185443 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52839005096785 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10347875 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50649361555968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52925053095936 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 814803551232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6383628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180499456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 % _timestamp‚1.7843687077673242e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3144568 _step‚110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10347877 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 649.211287648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9528193 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 843014917632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6383625 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53739856647168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13185443 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3144806 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53739751671808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23533318 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 811827995473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 814715615232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10347875 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205403136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10347877 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53739856647168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52027194140672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13195418 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13195418 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 811914132555 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52925036056576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 843195417088 trainer/global_step‚1399 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52925053095936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52925036056576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 811914132555 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52925036056576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 843195417088 trainer/global_step‚1399 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52925053095936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52925036056576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53739751671808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 814715615232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23543295 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13185443 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52839005096785 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10347875 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50649361555968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52925053095936 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 814803551232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6383628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180499456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 % _timestamp‚1.7843687077673242e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3144568 _step‚110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10347877 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 649.211287648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9528193 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 843014917632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6383625 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53739856647168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13185443 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3144806 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53739751671808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23533318 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 811827995473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 814715615232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 C;DeviceStatsMonitor.on_train_batch_startÄŽI/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10347875 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205403136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10347877 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53739856647168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 189733376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52027194140672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13195418 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13195418 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52962164060160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49841052749824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13194724 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52962164060160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 815372917760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50684651988480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10355096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52063666176000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 815372917760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9534755 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843687081212842e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10355098 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6388109 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚1399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10355096 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9534990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209590272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13204713 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23559811 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52962147020800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3146878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23549820 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 843599238656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9534755 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843687081212842e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10355098 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 189733376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6388109 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚1399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10355096 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9534990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209590272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13204713 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23559811 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52962147020800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3146878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23549820 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 843599238656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52962164060160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49841052749824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13194724 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52962164060160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 815372917760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50684651988480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10355096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52063666176000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 815372917760oú‘×0å+ _wandbruntime‚649  _runtime‚649t­æü /loss/FRt1/assignment_loss‚3.002282495474078 /loss/FRt2/assignment_loss‚1.419814269694071 1loss/SRqqt1/assignment_loss‚1.102487737893081 2loss/SRqqt2/assignment_loss‚0.6928897736959586 2loss/FBt1/assignment_loss‚0.044513663910038304 0loss/FBt2/assignment_loss‚0.0884321701435266 _step‚112 0loss/SRqqt1/detection_loss‚1.397312053158635 1loss/SRqqt2/detection_loss‚0.7522221642894952 /loss/FBt2/detection_loss‚0.6054653746226953 &loss/total_loss‚12.942047819501799 epoch‚1 /loss/FRt1/detection_loss‚0.6207239835520069 /loss/FRt2/detection_loss‚1.3258860592262833 trainer/global_step‚1399 % _timestamp‚1.7843687081226864e+09 /loss/FBt1/detection_loss‚1.8900180738419294 _runtime‚ 649.566307388ú Dç€ /loss/FRt2/assignment_loss‚1.419814269694071 1loss/SRqqt1/assignment_loss‚1.102487737893081 2loss/SRqqt2/assignment_loss‚0.6928897736959586 2loss/FBt1/assignment_loss‚0.044513663910038304 0loss/FBt2/assignment_loss‚0.0884321701435266 _step‚112 0loss/SRqqt1/detection_loss‚1.397312053158635 1loss/SRqqt2/detection_loss‚0.7522221642894952 /loss/FBt2/detection_loss‚0.6054653746226953 &loss/total_loss‚12.942047819501799 epoch‚1 /loss/FRt1/detection_loss‚0.6207239835520069 /loss/FRt2/detection_loss‚1.3258860592262833 trainer/global_step‚1399 % _timestamp‚1.7843687081226864e+09 /loss/FBt1/detection_loss‚1.8900180738419294 _runtime‚ 649.56630738 /loss/FRt1/assignment_loss‚3.002282495474078p3Ješ‘è‚(:† ÎœíÒØå¦ gpu.0.gpu‚61.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äŽY±¯é‚(:¤ ÎœíÒà€øŽ" disk.dm-0.out‚2134.05224609375 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 817.62890625% disk.nvme0n1.out‚2134.05224609375* proc.memory.percent‚16.155457496643066 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚18.375# proc.memory.rssMB‚ 2646.91015625 disk.dm-0.in‚ 817.62890625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚18.375) disk./.usagePercent‚39.60419248587985 disk.drbd2165.out‚0 disk.rbd1.out‚1.625" disk.nvme0c0n1.in‚ 817.62890625 network.recv‚255840 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚2134.05224609375$ memory_percent‚97.33142852783203 cpu‚26.669092143086246 proc.cpu.threads‚17 network.sent‚ 1.630219e+06$ disk./.usageGB‚590.1310882568359 disk.dm-1.in‚0 disk.dm-1.out‚18.375 disk.sda3.in‚0% proc.memory.availableMB‚ 437.21875”{Zl0ê+ _wandbruntime‚671  _runtime‚671´×ÊÆëÀ % _timestamp‚1.7843687301814172e+09 _runtime‚ 671.626613635 _step‚113 & lr-AdamW/pg1‚0.0008474904722080357 & lr-AdamW/pg2‚0.0008474904722080357 trainer/global_step‚1449:ëÊìÄ & lr-AdamW/pg2‚0.0008474904722080357 trainer/global_step‚1449 % _timestamp‚1.7843687301814172e+09 _runtime‚ 671.626613635 _step‚113 & lr-AdamW/pg1‚0.0008474904722080357qîMD©0í+ _wandbruntime‚671  _runtime‚671.w@Cî«O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52424073263104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178402304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10708925 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 843303776256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6607828 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24360270 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3252441 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51551716966400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55623800108032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13661320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54780601307136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚1449 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3252668 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚ 671.626698445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54691004842893 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 % _timestamp‚1.7843687301821837e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13651345 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13661320 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9860496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55623800108032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54780584267776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10708927 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 872356296704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53850795907072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53850778867712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13651345 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9860266 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55623905083392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 872534699008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _step‚“ãn 114 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 843215840256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 840312112263 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10708927 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55623905083392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚99750ÏÒµ'ï¯O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 840312112263 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10708927 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55623905083392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52424073263104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178402304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10708925 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 843303776256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6607828 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24360270 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3252441 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51551716966400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55623800108032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13661320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54780601307136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚1449 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3252668 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚ 671.626698445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54691004842893 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 % _timestamp‚1.7843687301821837e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13651345 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13661320 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9860496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55623800108032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54780584267776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10708927 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 872356296704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53850795907072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53850778867712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13651345 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9860266 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55623905083392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 872534699008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _step‚114 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 843215840256rᓧb0ð+ _wandbruntime‚671  _runtime‚671~áÍ%»&ñµM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55661585414144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55661480431616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184686592 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24386763 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55661585414144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52459567452160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54728147376305 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13670615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 671.969393789 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 840879433905 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54817695232000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 873123657728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24386763 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 843785199616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3254585 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 872938971136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53887250903040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10716146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52459777042432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6612312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚115 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10716148 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9866894 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13660626 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24376772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54728044199751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13660626 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54817712271360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10716148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53887267942400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13670615 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 843873142784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6612309 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51586628481024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚1449 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 840793296711 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54817695232000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 873123657728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24386763 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 843785199616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3254585 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 872938971136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53887250903040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10716146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52459777042432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6612312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚115 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10716148 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9866894 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13660626 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24376772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54728044199751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13660626 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54817712271360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10716148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53887267942400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13670615 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 843873142784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6612309 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51586628481024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚1449 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 840793296711 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55661585414144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55661480431616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184686592 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24386763 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55661585414144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52459567452160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54728147376305 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13670615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 671.969393789 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 UHDeviceStatsMonitÌYMÕðor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 840879433905 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146sNŽ,Ù0ó+ _wandbruntime‚671  _runtime‚671ñèÐIôÿ % _timestamp‚1.7843687305269234e+09 0loss/FRt1/assignment_loss‚3.0918786111197254 2loss/SRqqt1/assignment_loss‚0.9276067418112819 1loss/FBt2/assignment_loss‚0.12005396235284829 /loss/FRt1/detection_loss‚0.6446119923002894 epoch‚1 _runtime‚ 671.970489497 _step‚116 /loss/FBt1/detection_loss‚1.4735724275762385 trainer/global_step‚1449 0loss/FRt2/assignment_loss‚1.6625903250994505 2loss/SRqqt2/assignment_loss‚0.6687207860407224 /loss/FBt1/assignment_loss‚0.104400045376017 1loss/SRqqt2/detection_loss‚0.7985327852199252 /loss/FRt2/detection_loss‚1.4188166990324302 1loss/SRqqt1/detection_loss‚1.3979961987850922 /loss/FBt2/detection_loss‚0.5769983052804664 &loss/total_loss‚12.885778879994486Ž†ˆ3 õƒ 0loss/FRt1/assignment_loss‚3.0918786111197254 2loss/SRqqt1/assignment_loss‚0.9276067418112819 1loss/FBt2/assignment_loss‚0.12005396235284829 /loss/FRt1/detection_loss‚0.6446119923002894 epoch‚1 _runtime‚ 671.970489497 _step‚116 /loss/FBt1/detection_loss‚1.4735724275762385 trainer/global_step‚1449 0loss/FRt2/assignment_loss‚1.6625903250994505 2loss/SRqqt2/assignment_loss‚0.6687207860407224 /loss/FBt1/assignment_loss‚0.104400045376017 1loss/SRqqt2/detection_loss‚0.7985327852199252 /loss/FRt2/detection_loss‚1.4188166990324302 1loss/SRqqt1/detection_loss‚1.3979961987850922 /loss/FBt2/detection_loss‚0.5769983052804664 &loss/total_loss‚12.885778879994486 % _timestamp‚1.7843687305269234e+09t+Fœö‚(:‘ ÝœíÒ†‰°‹ gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚166.474' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.294799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú²†¥»÷‚(:° ÝœíÒÞ㇎ disk.sda3.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0# memory_percent‚97.3360538482666 cpu‚26.65914749021357 disk.sda3.out‚ 18.57421875! disk.dm-0.out‚2137.0380859375* proc.memory.percent‚16.155457496643066 network.sent‚ 1.692849e+06$ disk./.usageGB‚590.1312217712402 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.625# proc.memory.rssMB‚ 2646.91015625 disk.sda.out‚ 18.57421875 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 18.57421875 disk.dm-0.in‚ 817.62890625) disk./.usagePercent‚39.60420144614355 disk.sda2.in‚0 network.recv‚261023$ disk.nvme0n1.out‚2137.0380859375 disk.rbd0.in‚0' proc.memory.availableMB‚ 436.4609375" disk.nvme0c0n1.in‚ 817.62890625& disk.nvme0c0n1.out‚2137.0380859375 disk.nvme0n1.in‚ 817.62890625 disk.sda1.out‚0 disk.dm-1.in‚0%æù'œø‚(:‘ ìœíÒ⯼Š gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚172.666' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.533199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g„ç¸ù‚(:­ ìœíÒµ³ÚŒ$ disk./.usageGB‚590.1312294006348& disk.nvme0c0n1.out‚2142.2080078125# proc.memory.rssMB‚ 2646.91015625 disk.sda2.out‚0 disk.dm-0.in‚ 817.62890625" disk.nvme0c0n1.in‚ 817.62890625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚1.6875$ memory_percent‚97.33691215515137* proc.memory.percent‚16.155457496643066 network.recv‚266078) disk./.usagePercent‚39.60420195815862 disk.dm-1.in‚0 disk.dm-1.out‚ 18.703125 disk.sda3.in‚0 disk.sda3.out‚ 18.703125$ disk.nvme0n1.out‚2142.2080078125' proc.memory.availableMB‚ 436.3203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 817.62890625 cpu‚26.649655880255278 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 18.703125 disk.rbd1.in‚0 network.sent‚ 1.755461e+06! disk.dm-0.out‚2142.2080078125]Íä„0ú+ _wandbruntime‚694  _runtime‚694veìUÆûÀ % _timestamp‚1.7843687527123945e+09 _runtime‚ 694.156150222 _step‚117 & lr-AdamW/pg1‚0.0008473990999786469 & lr-AdamW/pg2‚0.0008473990999786469 trainer/global_step‚1499XÀÏzÊüÄ % _timestamp‚1.7843687527123945e+09 _runtime‚ 694.156150222 _step‚117 & lr-AdamW/pg1‚0.0008473990999786469 & lr-AdamW/pg2‚0.0008473990999786469 trainer/global_step‚1499u'ûúA0ý+ _wandbruntime‚694  _runtime‚694Õÿ Þ³'þ­O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57507953519616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54199258422784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 901940821504 % _timestamp‚1.7843687527133963e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56636149518336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53297292697600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3360471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14127222 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10192499 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 868710091971 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 868623954889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 871716065280 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 871716065280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 694.157402186 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10192264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57507953519616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57507848544256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14117247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11069977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56636132478976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56636132478976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56636149518336 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 871804001280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11069975 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57507848544256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6832028àýLC·'ÿ±O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25197199 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 901758224896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57507848544256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14117247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11069977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56636132478976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56636132478976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56636149518336 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 871804001280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11069975 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57507848544256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6832028 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57507953519616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54199258422784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 901940821504 % _timestamp‚1.7843687527133963e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56636149518336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53297292697600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3360471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14127222 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10192499 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 868710091971 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 868623954889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 871716065280 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 871716065280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 694.157402186 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10192264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57507953519616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11077196 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 869277413613 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 872285424640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53332229115904 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 902524123648 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25213715 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6836509 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56580147122413 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3362629 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 902345728512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14126528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56673260482560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55710852669440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11077198 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11077198 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55710869708800 _step‚119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57545633850368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56673243443200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14136517 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56673260482560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14136517 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25203724 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _runtime‚ 694.503262116 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25213715 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11077196 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 872373367808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178395136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 trainer/global_step‚1499 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10198913 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6836512 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57545528867840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14126528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843687530592053e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57545633850368 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10198913 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6836512 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57545528867840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14126528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843687530592053e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57545633850368 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11077196 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 869277413613 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 872285424640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53332229115904 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 902524123648 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25213715 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6836509 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56580147122413 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3362629 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 902345728512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14126528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56673260482560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55710852669440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11077198 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11077198 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55710869708800 _step‚119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57545633850368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56673243443200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14136517 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56673260482560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14136517 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25203724 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _runtime‚ 694.503262116 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25213715 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11077196w[Úì0ƒ+ _wandbruntime‚694  _runtime‚6946!‘ý„þ /loss/FRt1/assignment_loss‚3.084149639554076 0loss/FRt2/assignment_loss‚1.3147560079585632 1loss/FBt1/assignment_loss‚0.07318756786795953 /loss/FRt2/detection_loss‚1.3207391074312038 /loss/FBt2/detection_loss‚0.5092571978708981 &loss/total_loss‚12.029272287410192 epoch‚1 % _timestamp‚1.7843687530609827e+09 2loss/SRqqt1/assignment_loss‚0.9361776889283333 2loss/SRqqt2/assignment_loss‚0.6171745106355908 1loss/FBt2/assignment_loss‚0.09658200279269429 1loss/SRqqt2/detection_loss‚0.7031198940277099 trainer/global_step‚1499 _runtime‚ 694.504701614 _step‚120 .loss/FRt1/detection_loss‚0.683098552996807 1loss/SRqqt1/detection_loss‚1.4082541281715135 .loss/FBt1/detection_loss‚1.282775989174843Î÷œ…‚ 2loss/SRqqt1/assignment_loss‚0.9361776889283333 2loss/SRqqt2/assignment_loss‚0.6171745106355908 1loss/FBt2/assignment_loss‚0.09658200279269429 1loss/SRqqt2/detection_loss‚0.7031198940277099 trainer/global_step‚1499 _runtime‚ 694.504701614 _step‚120 .loss/FRt1/detection_loss‚0.683098552996807 1loss/SRqqt1/detection_loss‚1.4082541281715135 .loss/FBt1/detection_loss‚1.282775989174843 /loss/FRt1/assignment_loss‚3.084149639554076 0loss/FRt2/assignment_loss‚1.3147560079585632 1loss/FBt1/assignment_loss‚0.07318756786795953 /loss/FRt2/detection_loss‚1.3207391074312038 /loss/FBt2/detection_loss‚0.5092571978708981 &loss/total_loss‚12.029272287410192 epoch‚1 % _timestamp‚1.7843687530609827e+09xؽO‘†‚(:† ûœíҳܗ‹ gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚129.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ä3´¨‡‚(: ûœíÒÌèÍ$ memory_percent‚97.33552932739258 network.recv‚269813& disk.nvme0c0n1.out‚2147.0888671875 disk.dm-1.in‚0 disk.dm-1.out‚ 18.984375 cpu‚26.641745823253267 disk.dm-0.in‚887.125 disk.nvme0c0n1.in‚887.125$ disk./.usageGB‚590.1313858032227 disk.sda3.out‚ 18.984375 disk.sda.in‚0 disk.sda.out‚ 18.984375 disk.rbd1.in‚0 disk.rbd1.out‚1.6875 network.sent‚ 1.760435e+06 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚2147.0888671875# proc.memory.rssMB‚ 2646.91015625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.out‚0! disk.dm-0.out‚2147.0888671875 disk.nvme0n1.in‚887.125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) disk./.usagePercent‚39.60421245446752 disk.sda1.in‚0& proc.memory.availableMB‚ 436.546875* proc.memory.percent‚16.155457496643066Bí 0ˆ+ _wandbruntime‚716  _runtime‚716ÆEí`ʼn¿ & lr-AdamW/pg1‚0.0008473009466395404 & lr-AdamW/pg2‚0.0008473009466395404 trainer/global_step‚1549 $ _timestamp‚1.784368774941244e+09 _runtime‚ 716.385081305 _step‚121Ö¨<ÉŠÃ & lr-AdamW/pg1‚0.0008473009466395404 & lr-AdamW/pg2‚0.0008473009466395404 trainer/global_step‚1549 $ _timestamp‚1.784368774941244e+09 _runtime‚ 716.385081305 _step‚121y?_f·0‹+ _wandbruntime‚716  _runtime‚716!Ç Â³'Œ­O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170013696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57497982400512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 900304226304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14583149 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 931165749760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58395107511551 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7056228 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55042868428800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59391896980480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58491697729536 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 900216290304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 900216290304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11431027 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10525711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58395004335109 trainer/global_step‚1549 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3469483 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194917376 _runtime‚ 716.389612356 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11431025 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58491680690176 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 900304226304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55974229095936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55042893332480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59392001955840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 931335763456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3469262 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55974034178560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59392001955840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14593124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7056225 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 897021934597 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14593124 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 % _timestamp‚1.7843687749423363e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14583149 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59391896980480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58491697729536 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11431025ýÁM;>±O VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 900216290304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 900216290304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11431027 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10525711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58395004335109 trainer/global_step‚1549 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3469483 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194917376 _runtime‚ 716.389612356 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11431025 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58491680690176 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 900304226304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55974229095936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55042893332480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59392001955840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 931335763456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668599 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3469262 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55974034178560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59392001955840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14593124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7056225 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 897021934597 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14593124 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 WJDeviceStatsMoni ¯&sytor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 % _timestamp‚1.7843687749423363e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14583149 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59391896980480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58491697729536 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11431025 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170013696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57497982400512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 900304226304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14583149 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 931165749760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58395107511551 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7056228 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55042868428800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59391896980480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58491697729536zŠ 0Ž+ _wandbruntime‚716  _runtime‚716]Q¼è¼&¶M G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26040667 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14602419 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58432146868521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 931925422592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167909376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10532086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11438246 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59429682286592 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 900785649664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192813056 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7060709 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3471377 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55077804847104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58528791654400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56009537456640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 897589256127 $ _timestamp‚1.784368775286296e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59429682286592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 900873592832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11438248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3471585 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14592430 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26040667 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11438248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 900873592832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 897675393321 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57534471475200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 931757513216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128TÖWÒÀ&ºM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11438246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58528808693760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 900785649664 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59429577304064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58528791654400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10532297 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14602419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58528808693760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 931757513216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26040667 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14602419 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58432146868521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176554 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 931925422592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167909376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10532086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11438246 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59429682286592 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 900785649664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192813056 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7060709 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3471377 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55077804847104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58528791654400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56009537456640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 897589256127 $ _timestamp‚1.784368775286296e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137194 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59429682286592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 900873592832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11438248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3471585 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14592430 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26040667 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11438248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669120 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 900873592832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943168 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 897675393321 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57534471475200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 925506051387 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15049051 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56788469063680 _runtime‚ 738.945689084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11792075 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15049051 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3577972 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 960822587392 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10858656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60247107257659 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60347245940736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 928716515328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7280425 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 928804451328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59321601206272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/inacUºm÷tive_split.all.peak‚677 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60347228901376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15059026 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11792075 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7280428 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26841126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59321584166912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61275945416704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15059026 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60247004081217 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 925419914305 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 928804451328 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _step‚126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60347245940736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26851103 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60347228901376 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61276050392064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 I½žf·'±O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7280428 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26841126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59321584166912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61275945416704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15059026 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60247004081217 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 925419914305 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 928804451328 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _step‚126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60347245940736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26851103 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60347228901376 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61276050392064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 925506051387 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15049051 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56788469063680 _runtime‚ 738.945689084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11792075 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15049051 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3577972 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 960822587392 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10858656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60247107257659 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572135 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137082 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60347245940736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 928716515328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7280425 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176442 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 928804451328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59321601206272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60347228901376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15059026 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11792075 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146~ËIwª0ž+ _wandbruntime‚739  _runtime‚739R×Í€½&Ÿ·M C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26867619 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚ 739.290176878 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 929285874688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3580138 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 925987235835 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61313625740288 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 929373817856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11799296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15058332 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961412340736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180492288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59358073241600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60384356904960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7284909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60284146614629 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7284912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 926073373029 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 961231848448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15068321 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10865288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60384339865600 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 929285874688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26857628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _step‚127 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 % _timestamp‚1.7843687978461142e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15058332 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3580376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57784587523072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11799296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60384356904960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10865047 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192ÔŽfB »M [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57784792919040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104982528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10865047 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26867619 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚ 739.290176878 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 929285874688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3580138 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 925987235835 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572135 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61313625740288 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 929373817856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11799296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15058332 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669120 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961412340736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180492288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59358073241600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60384356904960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7284909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60284146614629 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7284912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_äG bytes.small_pool.peak‚ 300375040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 926073373029 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 961231848448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15068321 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10865288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60384339865600 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 929285874688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26857628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _step‚127 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104982528 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 % _timestamp‚1.7843687978461142e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15058332 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3580376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57784587523072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11799296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60384356904960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5420199632644653 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07247607409954071 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6244262456893921 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6074387431144714 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7050057649612427 jet/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04351997375488281 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07697667181491852 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42500001192092896 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9866605401039124 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5821665525436401 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07402528077363968 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984694123268127 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07017987221479416 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08541224896907806 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4544270634651184 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09926833212375641 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 +jet/accuracy_3_of_3‚0.05700170248746872 *loss/val_total_loss‚11.859110598819433 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5741674900054932 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6874213218688965 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.72805255651474 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.059510450810194016 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6349785923957825 epoch‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862056970596313 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.72805255651474 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7152881026268005 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01352559681981802 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8360090851783752 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8727639317512512 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48924630880355835 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6731720566749573 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6176047325134277 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5379687547683716 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5243343710899353 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957873821258545 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.059991590678691864 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7112432718276978 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.984174907207489 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.056845732033252716 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.06860999017953873 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.48763394355773926 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7087718844413757 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.536683201789856 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9890614748001099 (particle/accuracy‚0.8350344896316528 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5821665525436401 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_1_of_5‚1 _step‚129 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3950617015361786 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912558794021606 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2430555373430252 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.67762690782547 *jet/accuracy_1_of_3‚0.8756220936775208 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06180712580680847 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.588255763053894 /particle/accuracy_3_of_4‚0.9833914637565613 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9846324920654297 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.31150496006011963 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 $ _timestamp‚1.784368818224666e+09 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0707462728023529 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10260417312383652 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5347222089767456 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3953988254070282 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868764281272888 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.8294117450714111 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09060131013393402ç(s2¥)¨ü{ ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.72805255651474 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7152881026268005 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01352559681981802 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8360090851783752 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8727639317512512 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48924630880355835 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6731720566749573 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6176047325134277 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5379687547683716 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9545454382896423 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5243343710899353 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957873821258545 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.059991590678691864 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7112432718276978 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.984174907207489 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.056845732033252716 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.06860999017953873 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.48763394355773926 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7087718844413757 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.536683201789856 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9890614748001099 (particle/accuracy‚0.8350344896316528 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5821665525436401 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_1_of_5‚1 _step‚129 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3950617015361786 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912558794021606 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2430555373430252 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.67762690782547 *jet/accuracy_1_of_3‚0.8756220936775208 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06180712580680847 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.588255763053894 /particle/accuracy_3_of_4‚0.9833914637565613 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9846324920654297 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.31150496006011963 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 $ _timestamp‚1.784368818224666e+09 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0707462728023529 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10260417312383652 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5347222089767456 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3953988254070282 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868764281272888 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.8294117450714111 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09060131013393402 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9946348071098328 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7046695351600647 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13815638422966003 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5763428807258606 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019427498802542686 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0020272601395845413 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5106934309005737 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06680779904127121 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988247811794281 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8346396088600159 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984694123268127 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6300398707389832 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05526025593280792 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.5369406938552856 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.051446445286273956 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924041032791138 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6217947602272034 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05784936249256134 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48819777369499207 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010296841152012348 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01071428507566452 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42727530002593994 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.09565310180187225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5292872190475464 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.600105881690979 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05996830016374588 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.711298942565918 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6815788149833679 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0690126121044159 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12043482065200806 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5961000323295593 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6226460337638855 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 trainer/global_step‚1601 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05912000313401222 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5739959478378296 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.11534769088029861 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9933598637580872 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33513981103897095 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5686083436012268 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9724112153053284 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 +jet/accuracy_1_of_1‚0.24777767062187195 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9165540337562561 7validation_average_jet_accuracy‚0.29785144329071045 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4993750751018524 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988612949848175 particle/accuracy_2_of_2‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6309687495231628 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4422324001789093 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05927557498216629 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7599999904632568 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5537364482879639 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5760366916656494 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5288689136505127 +jet/accuracy_2_of_3‚0.42356717586517334 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6971611380577087 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945505261421204 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.09066741913557053 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7599999904632568 'particle/f_score‚0.5829249620437622 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1760973185300827 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953663945198059 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13797065615653992 9"PÁé]urity/1FRt1SRqqt1FBt/event_purity‚0.2205163836479187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0011206032941117883 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 particle/accuracy_4_of_5‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6928103566169739 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_5_of_5‚0 _runtime‚ 759.669130779 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.35824260115623474 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007391300052404404 jet/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.07640392333269119 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.334700345993042 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_2‚0.7190206050872803 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21772228181362152 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3031553030014038 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.056845732033252716 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7006827592849731 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964591264724731 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 jet/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.914364218711853 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05447914078831673 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.0732741579413414 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9545454382896423 *jet/accuracy_2_of_4‚0.6526772975921631 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6961537599563599 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990265369415283 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.13552607595920563 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4555857181549072 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 particle/accuracy_1_of_1‚1 )jet/accuracy_1_of_4‚0.944498598575592 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.051446445286273956 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8965675830841064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934183955192566 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9868813753128052 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5420199632644653 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07247607409954071 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6244262456893921 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6074387431144714 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7050057649612427 jet/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04351997375488281 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07697667181491852 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42500001192092896 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9866605401039124 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5821665525436401 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07402528077363968 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984694123268127 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07017987221479416 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08541224896907806 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4544270634651184 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09926833212375641 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 +jet/accuracy_3_of_3‚0.05700170248746872 *loss/val_total_loss‚11.859110598819433 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5741674900054932 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6874213218688965 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.72805255651474 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.059510450810194016 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6349785923957825 epoch‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862056970596313ÓFOÝ‘©‚(:† ·íÒÆÍîŠ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å:\´ª‚(:© ·íÒé•ùŒ network.recv‚287659! disk.nvme0c0n1.in‚ 955.3515625 disk.rbd0.in‚0 disk.dm-1.out‚ 20.02734375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 955.3515625$ disk./.usageGB‚590.1943969726562 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 1.952144e+06% disk.nvme0c0n1.out‚2229.919921875# disk.nvme0n1.out‚2229.919921875 disk.dm-0.out‚2229.919921875 disk.rbd1.out‚1.875 disk.sda1.in‚0 disk.sda.out‚ 20.02734375 disk.drbd2165.out‚0# memory_percent‚97.7975606918335( proc.memory.availableMB‚ 360.84765625) proc.memory.percent‚16.22462272644043 cpu‚26.707386947200792 disk.sda2.out‚0( disk./.usagePercent‚39.6084411869181 disk.rbd0.out‚0 disk.sda3.out‚ 20.02734375 disk.sda3.in‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2658.2421875 disk.nvme0n1.in‚ 955.3515625 disk.sda1.out‚0 disk.sda2.in‚0ðîœé‘«‚(:† ÆíÒÓô¬Š gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚178.033' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g8Ý®¶¬‚(:« ÆíÒ’ºÍŒ disk.dm-0.in‚ 955.3515625 disk.drbd2165.out‚0 disk.dm-1.out‚ 20.3515625 cpu‚26.698916380341178 disk.sda2.in‚0% proc.memory.availableMB‚ 360.84375 network.sent‚ 2.009632e+06) disk./.usagePercent‚39.60841865825509 disk.sda3.in‚0 disk.sda3.out‚ 20.3515625% disk.nvme0n1.out‚2298.51123046875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚97.7975845336914 disk.drbd2165.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚2298.51123046875 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚292261 disk.sda.in‚0 disk.rbd1.out‚1.9375! disk.nvme0c0n1.in‚ 955.3515625 disk.sda2.out‚0) proc.memory.percent‚16.22462272644043$ disk./.usageGB‚590.1940612792969 disk.sda.out‚ 20.3515625 disk.sda1.out‚0" proc.memory.rssMB‚ 2658.2421875" disk.dm-0.out‚2298.51123046875 disk.nvme0n1.in‚ 955.3515625L¯1ž0­+ _wandbruntime‚782  _runtime‚782•¢L*Æ®À & lr-AdamW/pg2‚0.0008470843030292626 trainer/global_step‚1649 % _timestamp‚1.7843688405883512e+09 _runtime‚ 782.032141889 _step‚130 & lr-AdamW/pg1‚0.0008470843030292626ŠfM˯Š& lr-AdamW/pg1‚0.0008470843030292626 & lr-AdamW/pg2‚0.0008470843030292626 trainer/global_step‚1649 % _timestamp‚1.7843688405883512e+09 _runtime‚ 782.032141889 _step‚130‚CqÒU0°+ _wandbruntime‚782  _runtime‚782ïï+Xµ'±¯O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7609205 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28046179 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63163756466176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚1649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64137487061504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 973818531840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28056157 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62102420846592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15658526 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63163773505536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12387655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1006894795776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12387655 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 782.033364147 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15658526 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11369276 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63163756466176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3759824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 % _timestamp‚1.7843688405893922e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64137592037376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60455258098176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59448533315584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 970392632537 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64137487061504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11369029 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12387653 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7609208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1006724782592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 970306495451 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63072710302683 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62102403807232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59448558219264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63163773505536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 973818531840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15668502 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192~¿êõº'²´O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63163756466176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3759824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 % _timestamp‚1.7843688405893922e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64137592037376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60455258098176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59448533315584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 970392632537 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64137487061504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11369029 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12387653 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7609208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1006724782592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 970306495451 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63072710302683 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62102403807232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59448558219264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63163773505536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 973818531840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15668502 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7609205 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28046179 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63163756466176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚1649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64137487061504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 973818531840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28056157 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62102420846592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15658526 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63163773505536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12387655 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1006894795776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12387655 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 782.033364147 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15658526 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11369276 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473ƒìlYî0³+ _wandbruntime‚782  _runtime‚782Rí9u,´¹M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60490955343872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63200867430400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12394874 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63200884469760 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3762345 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63200884469760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59483469733888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 974299954688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11376037 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1007485609984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28072673 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60490764628480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62138875842560 _step‚132 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 974387898368 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63109852836099 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63109749659541 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 970873816981 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15667807 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843688409310186e+09 _runtime‚ 782.375216648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11375811 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚1649 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7613689 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15677797 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12394874 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190715392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62138892881920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936p9Ä&µ¾M J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15667807 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28072673 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60490764628480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62138875842560 _step‚132 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 974387898368 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63109852836099 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63109749659541 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 970873816981 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15667807 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843688409310186e+09 _runtime‚ 782.375216648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11375811 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚1649 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7613689 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15677797 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12394874 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190715392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62138892881920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60490955343872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63200867430400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12394874 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63200884469760 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3762345 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63200884469760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59483469733888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 974299954688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11376037 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1007485609984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66021535497728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65019321716736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3869867 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66021535497728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1002230820352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63926005573632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843688634658508e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63926022612992 _step‚135 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62230295475712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚1699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16124428 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11703491 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998790612245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66021640473600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16124428 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61194109046784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1036186428928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1036360636416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12748705 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16134404 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61194133950464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65019321716736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65019304677376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚ 804.909870487 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66021535497728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65019321716736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3869867 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66021535497728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1002230820352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63926005573632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843688634658508e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63926022612992 _step‚135 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62230295475712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚1699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16124428 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11703491 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998790612245 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66021640473600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16124428 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61194109046784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1036186428928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1036360636416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12748705 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16134404 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61194133950464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65019321716736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65019304677376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚ 804.909870487 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28889633 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66059320804352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1002800179712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12755924 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _step‚136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16133709 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999357933887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12755926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11710117 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16143699 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7837892 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036774825472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28899625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚1699 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3872001 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63962477608960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11709890 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61229020561408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63962494648320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1002888123392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3872225 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62265998685184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65056415641600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64961852582207 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12755924 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1002888123392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65056432680960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16143699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11709890 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61229020561408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63962494648320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1002888123392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3872225 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62265998685184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65056415641600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64961852582207 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12755924 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1002888123392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65056432680960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16143699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28889633 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66059320804352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1002800179712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12755924 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _step‚136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16133709 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999357933887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12755926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11710117 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16143699 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7837892 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036774825472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28899625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚1699 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3872001 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ˆ¼® 0Ä+ _wandbruntime‚805  _runtime‚805ü’ ;Åÿ /loss/FRt1/detection_loss‚0.6730096949520822 epoch‚2 trainer/global_step‚1699 0loss/FRt1/assignment_loss‚2.8645331456229606 0loss/FBt1/assignment_loss‚0.2635845455448501 /loss/FBt1/detection_loss‚1.3441545756963582 /loss/FBt2/detection_loss‚0.6157912084379712 2loss/SRqqt1/assignment_loss‚0.8659496255485075 1loss/FBt2/assignment_loss‚0.07975860192282144 % _timestamp‚1.7843688638217728e+09 _runtime‚ 805.265397047 &loss/total_loss‚12.117236578024844 1loss/SRqqt2/assignment_loss‚0.594960916706121 /loss/FRt2/detection_loss‚1.4382341267650618 1loss/SRqqt1/detection_loss‚1.4537132070000682 1loss/SRqqt2/detection_loss‚0.7535373810704519 _step‚137 0loss/FRt2/assignment_loss‚1.1700095487575906cC¿ Æ„ 0loss/FBt1/assignment_loss‚0.2635845455448501 /loss/FBt1/detection_loss‚1.3441545756963582 /loss/FBt2/detection_loss‚0.6157912084379712 2loss/SRqqt1/assignment_loss‚0.8659496255485075 1loss/FBt2/assignment_loss‚0.07975860192282144 % _timestamp‚1.7843688638217728e+09 _runtime‚ 805.265397047 &loss/total_loss‚12.117236578024844 1loss/SRqqt2/assignment_loss‚0.594960916706121 /loss/FRt2/detection_loss‚1.4382341267650618 1loss/SRqqt1/detection_loss‚1.4537132070000682 1loss/SRqqt2/detection_loss‚0.7535373810704519 _step‚137 0loss/FRt2/assignment_loss‚1.1700095487575906 /loss/FRt1/detection_loss‚0.6730096949520822 epoch‚2 trainer/global_step‚1699 0loss/FRt1/assignment_loss‚2.8645331456229606‰'fÇ‘Ç‚(:† äíÒÒ‰”‹ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚186.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|aPy°È‚(:¥ äíÒ‡®ËŒ network.sent‚ 2.077405e+06 disk.rbd1.out‚2 disk.dm-1.out‚ 20.71484375 network.recv‚300441 disk.nvme0n1.in‚ 955.3515625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2658.2421875) proc.memory.percent‚16.22462272644043! disk.dm-0.out‚2308.3291015625$ disk./.usageGB‚590.1943969726562 disk.sda3.out‚ 20.71484375 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚2308.3291015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 cpu‚26.70343285024625 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-0.in‚ 955.3515625$ memory_percent‚97.79834747314453% proc.memory.availableMB‚ 360.71875( disk./.usagePercent‚39.6084411869181 disk.sda3.in‚0 disk.sda.out‚ 20.71484375! disk.nvme0c0n1.in‚ 955.3515625$ disk.nvme0n1.out‚2308.3291015625 disk.dm-1.in‚0 disk.sda2.in‚0]|o‘É‚(:† óíÒ¶¡ëŠ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚180.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zdN¼Ê‚(:± óíÒ¶ÒÎ disk.rbd0.in‚0 disk.sda.out‚ 20.87890625 network.recv‚305954 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 1041.6328125 disk.sda2.out‚0 disk.dm-0.in‚ 1041.6328125 disk.sda.in‚0 cpu‚26.715326060637725 proc.cpu.threads‚17* disk./.usagePercent‚39.608462435543444 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚2312.43212890625$ memory_percent‚97.79694080352783" disk.nvme0c0n1.in‚ 1041.6328125 disk.dm-1.in‚0 disk.dm-1.out‚ 20.87890625( proc.memory.availableMB‚ 360.94921875' disk.nvme0c0n1.out‚2312.43212890625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 20.87890625$ disk./.usageGB‚590.1947135925293 disk.sda3.in‚0 network.sent‚ 2.140044e+06 disk.rbd1.in‚0" proc.memory.rssMB‚ 2658.2421875) proc.memory.percent‚16.22462272644043 disk.rbd1.out‚2% disk.nvme0n1.out‚2312.43212890625¿ØP0Ë+ _wandbruntime‚827  _runtime‚827<4µ½ÅÌ¿ & lr-AdamW/pg1‚0.0008468405552570955 & lr-AdamW/pg2‚0.0008468405552570955 trainer/global_step‚1749 % _timestamp‚1.7843688863516605e+09 _runtime‚ 827.79564732 _step‚138{™HÆÊÍÄ _step‚138 & lr-AdamW/pg1‚0.0008468405552570955 & lr-AdamW/pg2‚0.0008468405552570955 trainer/global_step‚1749 % _timestamp‚1.7843688863516605e+09 _runtime‚ 827.79564732Š½4G0Î+ _wandbruntime‚827  _runtime‚827¹~ ƒrϵO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13109753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1065849316864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1065679303680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65749624379392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16590330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1030818981888 T ÖgªI$GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16590330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12038728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66874852888576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8057608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67905688909824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66874852888576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3981120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62939684777984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13109755 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66874869927936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67905688909824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64005364081664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13109755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 % _timestamp‚1.7843688863530152e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1030731045376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67905583933952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8057605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚1749 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65749607340032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62939709681664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1027102454867 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0‹ÉæÀ'кO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67905688909824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66874852888576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3981120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62939684777984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13109755 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66874869927936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67905688909824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64005364081664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13109755 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 % _timestamp‚1.7843688863530152e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1030731045376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67905583933952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8057605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚1749 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65749607340032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62939709681664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1027102454867 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13109753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1065849316864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1065679303680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65749624379392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16590330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1030818981888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16590330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12038728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66874852888576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8057608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‹•þ0Ñ+ _wandbruntime‚828  _runtime‚828Ï®m“Å&Ò¿M % _timestamp‚1.7843688867141454e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1066440184320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67943369240576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3983371 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62974621196288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1066272275456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66911963852800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66813852328315 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13116974 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29716585 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13116976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13116976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1031388348416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16609601 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13116974 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12045240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1031300404736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67943369240576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1031300404736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29726577 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66911980892160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12045463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062089 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64040868568064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062092 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3983151 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29726577 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64041061380608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62974596292608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936—5¥íß ÓÄM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65786079375360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13116976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13116976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1031388348416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16609601 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13116974 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12045240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1031300404736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 VCDeviceStatsMonitor.on_train_batch_”²aëend/allocated_bytes.all.allocated‚67943369240576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1031300404736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29726577 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66911980892160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12045463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062089 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64040868568064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062092 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3983151 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29726577 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64041061380608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62974596292608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 % _timestamp‚1.7843688867141454e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1066440184320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67943369240576 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3983371 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62974621196288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1066272275456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66911963852800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66813852328315 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13116974 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29716585 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8281808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65780436315136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4091815 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67573226145792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69789632370176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65780639620608 trainer/global_step‚1799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69789737346048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68730401099776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13470803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13470803 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64685260509184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30527035 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1059319206912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1059231270400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30537013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13470805 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68628812717453 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1059231270400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17056232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1055586571661 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1055500434575 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8281805 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68730401099776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1095354207744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68628709541007 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1095175805952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17056232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67573209106432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68730418139136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13470805 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086M¤À'ÞºO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13470805 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68628812717453 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1059231270400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17056232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1055586571661 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1055500434575 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8281805 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68730401099776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1095354207744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68628709541007 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1095175805952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17056232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67573209106432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68730418139136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13470805 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1059319206912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8281808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65780436315136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4091815 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67573226145792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69789632370176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65780639620608 trainer/global_step‚1799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69789737346048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68730401099776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13470803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13470803 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64685260509184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30527035 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1059319206912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1059231270400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30537013 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239Ã17u0ß+ _wandbruntime‚850  _runtime‚850þáJà¿M UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1059888573440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8286292 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30543537 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30553529 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12380025 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13478024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13478024 RpI× 5EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1056067756105 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1056153893303 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30553529 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68767512064000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13478026 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69827417676800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17075503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30543537 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17065513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4093961 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17065513 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67609681141760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68767512064000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1059800629760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1095941143040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1059888573440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12380253 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68665748897865 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13478026 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17075503 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8286289 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1059800629760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64720172023808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67609698181120 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚1799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68767529103360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65816138070528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 850.946134045 EÊ&áÄM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1059800629760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64720172023808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67609698181120 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚1799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68767529103360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65816138070528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 850.946134045 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1059888573440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8286292 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30543537 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30553529 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12380025 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13478024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13478024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1056067756105 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1056153893303 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30553529 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68767512064000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13478026 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69827417676800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17075503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30543537 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17065513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4093961 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17065513 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67609681141760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68767512064000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1059800629760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1095941143040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1059888573440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12380253 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68665748897865 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13478026 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17075503 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8286289 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936潟ü0â+ _wandbruntime‚850  _runtime‚850owfâãþ _runtime‚ 850.947685064 1loss/FBt1/assignment_loss‚0.06258699905083631 /loss/FRt1/detection_loss‚0.6029172198511332 /loss/FRt2/detection_loss‚1.3525011914298926 &loss/total_loss‚11.263966049519581 % _timestamp‚1.7843689095040543e+09 0loss/FRt2/assignment_loss‚1.3374426358125426 2loss/SRqqt1/assignment_loss‚0.8555113965035481 2loss/SRqqt2/assignment_loss‚0.6184463312930387 /loss/FBt2/assignment_loss‚0.015192062534868 1loss/SRqqt1/detection_loss‚1.1748854297042732 1loss/SRqqt2/detection_loss‚0.8051374038573409 /loss/FBt2/detection_loss‚0.6563000055629975 epoch‚2 /loss/FRt1/assignment_loss‚2.475638429696361 /loss/FBt1/detection_loss‚1.3074069442227483 _step‚145 trainer/global_step‚1799 Ÿ,¼ äƒ 1loss/FBt1/assignment_loss‚0.06258699905083631 /loss/FRt1/detection_loss‚0.6029172198511332 /loss/FRt2/detection_loss‚1.3525011914298926 &loss/total_loss‚11.263966049519581 % _timestamp‚1.7843689095040543e+09 0loss/FRt2/assignment_loss‚1.3374426358125426 2loss/SRqqt1/assignment_loss‚0.8555113965035481 2loss/SRqqt2/assignment_loss‚0.6184463312930387 /loss/FBt2/assignment_loss‚0.015192062534868 1loss/SRqqt1/detection_loss‚1.1748854297042732 1loss/SRqqt2/detection_loss‚0.8051374038573409 /loss/FBt2/detection_loss‚0.6563000055629975 epoch‚2 /loss/FRt1/assignment_loss‚2.475638429696361 /loss/FBt1/detection_loss‚1.3074069442227483 _step‚145 trainer/global_step‚1799 _runtime‚ 850.947685064‘íÔ6‘å‚(:† ‘žíÒãÙêŠ gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚180.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼D ³æ‚(:¨ ‘žíÒŒ¾º disk.rbd1.out‚2.0625# disk.nvme0c0n1.out‚ 2322.0078125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 21.45703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚26.71906437429494 disk.dm-0.in‚ 1041.6328125 disk.dm-0.out‚ 2322.0078125 disk.sda.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1041.6328125 disk.sda2.in‚0$ memory_percent‚97.79942035675049) disk./.usagePercent‚39.60839510556195 network.recv‚314595 proc.cpu.threads‚17$ disk./.usageGB‚590.1937103271484 disk.drbd2165.out‚0 disk.dm-1.out‚ 21.45703125) proc.memory.percent‚16.22462272644043 network.sent‚ 2.207546e+06 disk.sda.out‚ 21.45703125 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 1041.6328125 disk.sda1.in‚0" proc.memory.rssMB‚ 2658.2421875! disk.nvme0n1.out‚ 2322.0078125 disk.sda1.out‚0( proc.memory.availableMB‚ 360.54296875=m+œç‚(:‘  žíÒƒšã‹ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚141.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.257600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r Í·è‚(:¬  žíÒÊݸ disk.drbd2165.out‚0 disk.sda3.out‚ 21.84765625$ memory_percent‚97.79906272888184 cpu‚26.71122690361075 disk.nvme0n1.in‚ 1041.6328125# disk.nvme0n1.out‚2326.763671875 disk.rbd0.in‚0 disk.dm-0.out‚2326.763671875) proc.memory.percent‚16.22462272644043 proc.cpu.threads‚17) disk./.usagePercent‚39.60845091520441 disk.rbd1.in‚0% disk.nvme0c0n1.out‚2326.763671875 disk.sda.in‚0$ disk./.usageGB‚590.1945419311523 disk.sda.out‚ 21.84765625 disk.rbd1.out‚2.125 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 21.84765625 disk.dm-0.in‚ 1041.6328125 network.recv‚319680 disk.sda2.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2658.2421875 network.sent‚ 2.270133e+06 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 1041.6328125' proc.memory.availableMB‚ 360.60156250„|Ø0é+ _wandbruntime‚872  _runtime‚872FˆY«ÆêÀ & lr-AdamW/pg2‚0.0008465697189394377 trainer/global_step‚1849 % _timestamp‚1.7843689315250113e+09 _runtime‚ 872.968804418 _step‚146 & lr-AdamW/pg1‚0.00084656971893943770püËëÅ % _timestamp‚1.7843689315250113e+09 _runtime‚ 872.968804418 _step‚146 & lr-AdamW/pg1‚0.0008465697189394377 & lr-AdamW/pg2‚0.0008465697189394377 trainer/global_step‚1849’€¤Ï0ì+ _wandbruntime‚872  _runtime‚872“Yîf»'íµO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 % _timestamp‚1.7843689315260444e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70585949310976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67555513437184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182333952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71673680806400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17532110 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71673680806400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1087819431936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70585949310976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70585966350336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67555695771136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12707234 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1087731495424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70480709287115 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17522134 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1087819431936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚147 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31353987 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69396827912192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69396810872832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128á3´îºO PCDeviceStatsMon =©’¨'itor.on_train_batch_start/active.small_pool.allocated‚17532110 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71673680806400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1087819431936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70585949310976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70585966350336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67555695771136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12707234 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1087731495424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70480709287115 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17522134 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1087819431936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚147 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31353987 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69396827912192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69396810872832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 % _timestamp‚1.7843689315260444e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70585949310976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67555513437184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182333952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71673680806400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0“á þÔ0ï+ _wandbruntime‚873  _runtime‚873{!—êÅ&ð¿M REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13839076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71711361129984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1088388798464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12713920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12714166 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13839076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199104000 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚71711466113024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4203674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70517851820531 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31380481 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1125259691008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70623077314560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66465747755008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69433299947520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67591206550016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _runtime‚ 873.316203698 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17541405 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17531415 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17531415 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70517748643973 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70623077314560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1125433891328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174200320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1088300854784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13839074 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13839074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8510492 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67591007446016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17541405 % _timestamp‚1.7843689318721519e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69433282908160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1088388798464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1084551873011 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70623060275200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2œõ@6Ê&ñÄM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4203431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17541405 % _timestamp‚1.7843689318721519e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69433282908160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1088388798464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1084551873011 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70623060275200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13839076 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71711361129984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1088388798464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12713920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12714166 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13839076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199104000 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚71711466113024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4203674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70517851820531 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31380481 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1125259691008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70623077314560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66465747755008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69433299947520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67591206550016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _runtime‚ 873.316203698 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17541405 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17531415 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17531415 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70517748643973 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70623077314560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1125433891328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174200320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1088300854784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13839074 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13839074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8510492 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67591007446016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73557834218496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13041535 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71220429678592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73557729242624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69330612941312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚1899 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68176436875264Ô¤ð\À'üºO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1112382531077 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174207488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68176411971584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17988036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14192903 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1154375177216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72441497522176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8730208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73557834218496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13041535 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71220429678592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73557729242624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69330612941312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚1899 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68176436875264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71220412639232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1116231720448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32190917 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4311562 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14192905 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17988036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚ 895.581789834—ñ~íÔ0ý+ _wandbruntime‚895  _runtime‚895 ¹©ˆÅ&þ¿M BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4313729 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18007307 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71256884674560 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 _runtime‚ 895.928678361 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72478608486400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72478608486400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1116889023488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13048421 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32207433 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17997317 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14200124 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1116889023488 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1116801079808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14200126 % _timestamp‚1.7843689544846077e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32197441 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73595514549248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73595409566208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69366316249088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚1899 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32197441 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73595514549248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32207433 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72478625525760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14200124 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18007307 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72369851566639 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14200126 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68211323486208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71256901713920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1116801079808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1154785270272 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72478625525760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13048200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4313511 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936«+ˆ^%ÿÄM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73595514549248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32207433 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72478625525760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14200124 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18007307 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72369851566639 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14200126 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68211323486208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71256901713920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1116801079808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1154785270272 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 QBDeviceStatsê6”@¥ Monitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72478625525760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13048200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4313511 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4313729 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18007307 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71256884674560 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 _runtime‚ 895.928678361 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72478608486400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72478608486400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1116889023488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13048421 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32207433 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17997317 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14200124 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1116889023488 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1116801079808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14200126 % _timestamp‚1.7843689544846077e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32197441 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73595514549248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73595409566208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69366316249088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚1899 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32197441 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0˜Í¦0€ + _wandbruntime‚895  _runtime‚895ZóàI ü /loss/FRt2/assignment_loss‚1.224352458000183 &loss/total_loss‚10.238855187773813 2loss/SRqqt1/assignment_loss‚0.8035567068563612 0loss/SRqqt2/assignment_loss‚0.65654793730163 2loss/FBt1/assignment_loss‚0.022543099381996747 /loss/FBt2/detection_loss‚0.5241902876358766 epoch‚2 % _timestamp‚1.7843689544864783e+09 _runtime‚ 895.930026606 _step‚153 1loss/FBt2/assignment_loss‚0.03535462059702796 /loss/FRt1/detection_loss‚0.6399022069791802 /loss/FRt2/detection_loss‚1.3471721824277192 /loss/FBt1/detection_loss‚0.6271065595344856 /loss/FRt1/assignment_loss‚2.453784933934609 1loss/SRqqt1/detection_loss‚1.2109727803992745 /loss/SRqqt2/detection_loss‚0.69337141472547 trainer/global_step‚1899ËR ‚  &loss/total_loss‚10.238855187773813 2loss/SRqqt1/assignment_loss‚0.8035567068563612 0loss/SRqqt2/assignment_loss‚0.65654793730163 2loss/FBt1/assignment_loss‚0.022543099381996747 /loss/FBt2/detection_loss‚0.5241902876358766 epoch‚2 % _timestamp‚1.7843689544864783e+09 _runtime‚ 895.930026606 _step‚153 1loss/FBt2/assignment_loss‚0.03535462059702796 /loss/FRt1/detection_loss‚0.6399022069791802 /loss/FRt2/detection_loss‚1.3471721824277192 /loss/FBt1/detection_loss‚0.6271065595344856 /loss/FRt1/assignment_loss‚2.453784933934609 1loss/SRqqt1/detection_loss‚1.2109727803992745 /loss/SRqqt2/detection_loss‚0.69337141472547 trainer/global_step‚1899 /loss/FRt2/assignment_loss‚1.224352458000183™mÓšœƒ ‚(:‘ ¾žíÒåÊ™ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚185.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÛU‡º„ ‚(:¯ ¾žíÒ„ÍûŽ disk.sda1.out‚0 disk.nvme0n1.in‚ 1111.6171875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1111.6171875" disk.dm-0.out‚2334.76318359375 network.recv‚328691 disk.sda.out‚ 22.42578125" disk.nvme0c0n1.in‚ 1111.6171875 disk.dm-1.out‚ 22.42578125" proc.memory.rssMB‚ 2658.2421875 disk.sda2.in‚0) disk./.usagePercent‚39.60835900849962' disk.nvme0c0n1.out‚2334.76318359375 proc.cpu.threads‚17 network.sent‚ 2.337891e+06# disk./.usageGB‚590.193172454834 disk.rbd1.out‚2.25$ memory_percent‚97.80206680297852 disk.drbd2165.in‚0 disk.drbd2165.out‚0& proc.memory.availableMB‚ 360.109375) proc.memory.percent‚16.22462272644043 disk.sda.in‚0 disk.sda1.in‚0 cpu‚26.69568106916132 disk.sda3.in‚0 disk.sda3.out‚ 22.42578125 disk.rbd1.in‚0% disk.nvme0n1.out‚2334.76318359375 disk.rbd0.in‚0C‹J‘… ‚(:† Íží҃ψ‹ gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚169.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÑÝtº† ‚(:¯ ÍžíÒÁžý‹ disk.rbd0.in‚0 disk.drbd2165.in‚0& proc.memory.availableMB‚ 360.109375" proc.memory.rssMB‚ 2658.2421875 proc.cpu.threads‚17" disk.dm-0.out‚2337.96435546875 network.sent‚ 2.400669e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 22.62890625 disk.rbd0.out‚0 disk.sda3.out‚ 22.62890625 disk.sda.in‚0' disk.nvme0c0n1.out‚2337.96435546875 disk.sda2.out‚0) proc.memory.percent‚16.22462272644043 disk.dm-1.in‚0% disk.nvme0n1.out‚2337.96435546875 disk.sda1.in‚0 disk.dm-1.out‚ 22.62890625 disk.drbd2165.out‚0# disk./.usageGB‚590.193775177002" disk.nvme0c0n1.in‚ 1111.6171875 disk.nvme0n1.in‚ 1111.6171875) disk./.usagePercent‚39.60839945769003 disk.dm-0.in‚ 1111.6171875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.25$ memory_percent‚97.80206680297852 cpu‚26.68797003735746 network.recv‚333636ø¢„á0‡ + _wandbruntime‚918  _runtime‚9187ÿšøƈ À _runtime‚ 918.064556669 _step‚154 & lr-AdamW/pg1‚0.0008462718114281933 & lr-AdamW/pg2‚0.0008462718114281933 trainer/global_step‚1949 % _timestamp‚1.7843689766205826e+09@J<ˉ Å & lr-AdamW/pg1‚0.0008462718114281933 & lr-AdamW/pg2‚0.0008462718114281933 trainer/global_step‚1949 % _timestamp‚1.7843689766205826e+09 _runtime‚ 918.064556669 _step‚154šâÕB 0Š + _wandbruntime‚918  _runtime‚918ñE»'‹ µO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75441777678848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69921987702784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 _step‚155 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1144731945472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33017869 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1183890221056 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74184708779331 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75441882654720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8954405 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18453938 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18463914 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13375186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4421014 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74297045733376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14553955 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8954408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1140694373699 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14553955 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1144819881984 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74297045733376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14553953 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843689766218672e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14553953 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74297062772736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71105902827520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1183707624960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71105695327744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73044031444992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75441882654720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74297062772736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73044014405632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140780510785 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182596096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75441777678848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18463914 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4420781 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69922012606464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040õä s$Œ ºO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1144731945472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 918.068953901 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74297045733376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14553953 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843689766218672e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14553953 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74297062772736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71105902827520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1183707624960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71105695327744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73044031444992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75441882654720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74297062772736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73044014405632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140780510785 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182596096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75441777678848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18463914 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4420781 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69922012606464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75441777678848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69921987702784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 _step‚155 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1144731945472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33017869 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1183890221056 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74184708779331 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75441882654720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8954405 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18453938 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18463914 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13375186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4421014 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74297045733376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14553955 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8954408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1140694373699 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14553955 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1144819881984 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 $ _timestamp‚1.784368976970731e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69956899217408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 918.414801299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33034385 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4422940 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74334173736960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14561174 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚1949 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1145301304832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _step‚156 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33034385 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33024393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14561176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69956924121088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141347832427 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74334156697600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13381829 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73080486440960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33024393 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚75479562985472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1145389248512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1145301304832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74221748136189 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74334173736960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8958892 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73080503480320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18473209 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75479562985472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1141261695229 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74334156697600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3“½®)É& ÃM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1145301304832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _step‚156 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33034385 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33024393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14561176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69956924121088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141347832427 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74334156697600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13381829 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73080486440960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33024393 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚75479562985472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1145389248512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1145301304832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74221748136189 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74334173736960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8958892 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73080503480320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18473209 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75479562985472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1141261695229 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74334156697600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4423160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18463219 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14561176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8958889 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71141195206144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 $ _timestamp‚1.784368976970731e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69956899217408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 918.414801299 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33034385 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4422940 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74334173736960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14561174 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚1949œ—=¿0 + _wandbruntime‚918  _runtime‚918}sqf‘ ÿ % _timestamp‚1.7843689769723735e+09 _runtime‚ 918.415910393 /loss/FRt1/assignment_loss‚2.300254319908713 2loss/SRqqt1/assignment_loss‚0.7856811588223012 0loss/FBt1/assignment_loss‚0.1438669242539058 /loss/FRt2/detection_loss‚1.5669785749124203 0loss/FRt2/assignment_loss‚1.2342018340085004 2loss/SRqqt2/assignment_loss‚0.6745883130077858 2loss/FBt2/assignment_loss‚0.028765426915719382 /loss/FBt1/detection_loss‚0.9187573644404228 &loss/total_loss‚11.010835711728204 0loss/SRqqt1/detection_loss‚1.418422871916894 1loss/SRqqt2/detection_loss‚0.7653442153743165 epoch‚2 _step‚157 /loss/FRt1/detection_loss‚0.6284790914183823 /loss/FBt2/detection_loss‚0.5454956167488403 trainer/global_step‚1949³ø˜Ë ’ „ 0loss/FRt2/assignment_loss‚1.2342018340085004 2loss/SRqqt2/assignment_loss‚0.6745883130077858 2loss/FBt2/assignment_loss‚0.028765426915719382 /loss/FBt1/detection_loss‚0.9187573644404228 &loss/total_loss‚11.010835711728204 0loss/SRqqt1/detection_loss‚1.418422871916894 1loss/SRqqt2/detection_loss‚0.7653442153743165 epoch‚2 _step‚157 /loss/FRt1/detection_loss‚0.6284790914183823 /loss/FBt2/detection_loss‚0.5454956167488403 trainer/global_step‚1949 % _timestamp‚1.7843689769723735e+09 _runtime‚ 918.415910393 /loss/FRt1/assignment_loss‚2.300254319908713 2loss/SRqqt1/assignment_loss‚0.7856811588223012 0loss/FBt1/assignment_loss‚0.1438669242539058 /loss/FRt2/detection_loss‚1.5669785749124203ôD‘ñ‘“ ‚(:† ÜžíÒÇžšŠ gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚175.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 A¢<§” ‚(:œ ÜžíÒðçŠ cpu‚26.68052447647872$ memory_percent‚97.80125617980957 disk.dm-0.in‚ 1111.6171875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2658.24609375 disk.drbd2165.in‚0$ disk.nvme0c0n1.out‚ 2343.87109375 proc.cpu.threads‚17( disk./.usagePercent‚39.6083828172003 disk.sda3.in‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 2343.87109375 network.recv‚336880 disk.sda.out‚22.8125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 360.2421875 disk.sda3.out‚22.8125 disk.sda.in‚0 network.sent‚ 2.4056e+06 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1111.6171875 disk.nvme0n1.in‚ 1111.6171875 disk.dm-1.out‚22.8125 disk.dm-0.out‚ 2343.87109375 disk.rbd1.out‚2.3125$ disk./.usageGB‚590.1935272216797 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.22464656829834ʬÔC0• + _wandbruntime‚940  _runtime‚940þ øÆ– À % _timestamp‚1.7843689990343406e+09 _runtime‚ 940.478128337 _step‚158 & lr-AdamW/pg1‚0.0008461127118717269 & lr-AdamW/pg2‚0.0008461127118717269 trainer/global_step‚1999NìCBË— Å & lr-AdamW/pg1‚0.0008461127118717269 & lr-AdamW/pg2‚0.0008461127118717269 trainer/global_step‚1999 % _timestamp‚1.7843689990343406e+09 _runtime‚ 940.478128337 _step‚158žÐÛ©0˜ + _wandbruntime‚940  _runtime‚940ZÆ,!™ µO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76152610983936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76152610983936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1213336668160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18919840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9178605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186528256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14915005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18919840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76036811701885 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71667588337664 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1173232170496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4529855 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1169178490493 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14915003 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13708242 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33834843 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77325826115072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76152593944576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚77325931090944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1213175043584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L¡C?FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1173232170496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76152593944576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14915003 trainer/global_step‚1999 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚77325931090944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9178608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77325826115072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843689990353835e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33844821 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976¤ø¼ÁÀ'š ºO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1173232170496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76152593944576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14915003 trainer/global_step‚1999 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚77325931090944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9178608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77325826115072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843689990353835e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33844821 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76152610983936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76152610983936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1213336668160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18919840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9178605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186528256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14915005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18919840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76036811701885 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71667588337664 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1173232170496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4529855 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1169178490493 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14915003 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13708242 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33834843 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77325826115072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76152593944576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚77325931090944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1213175043584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ŸÆ™0› + _wandbruntime‚940  _runtime‚940t¼ÀÅ&œ ¿M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4532099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13715191 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18929121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1173889473536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1173801529856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 940.816001867 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚219 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77363506438656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14922226 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76189721948160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14922224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1169659674937 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1213931869184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1173801529856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9183089 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9183092 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33851345 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 % _timestamp‚1.7843689993719232e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33861337 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33861337 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72916234714624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76073747882297 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18929121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76189721948160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71702474948608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚77363611421696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13714972 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677Ï4KÊ& ÄM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚219 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77363506438656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14922226 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76189721948160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14922224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1169659674937 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1213931869184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1173801529856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9183089 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9183092 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33851345 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 % _timestamp‚1.7843689993719232e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33861337 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33861337 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72916234714624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76073747882297 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18929121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76189721948160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71702474948608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚77363611421696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13714972 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4532099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13715191 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18929121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1173889473536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1173801529856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 940.816001867 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E¨0¨ + _wandbruntime‚962  _runtime‚962}@/}»'© µO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79209874551296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1197490333115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34661795 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15276055 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73413164068864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78008142155776 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74656005179392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79209979527168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1201820332032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚215 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159527424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34671773 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77888708271547 % _timestamp‚1.7843690214639914e+09 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19385742 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4638463 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚218 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1201732395520 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79209979527168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15276053 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1242681583104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77888811447993 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15276053 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9402808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚163 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19385742 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1201732395520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14041486 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4638678 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚ 962.908081174 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1197576470201 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19395718 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1201820332032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9402805 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1242841110528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76691217938432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ІÖÀ'ª ºO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 184431104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19395718 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78008142155776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79209874551296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19385742 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4638463 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚218 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1201732395520 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79209979527168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15276053 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1242681583104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77888811447993 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15276053 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9402808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚163 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19385742 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1201732395520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14041486 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4638678 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚ 962.908081174 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1197576470201 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19395718 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1201820332032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9402805 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1242841110528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76691217938432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79209874551296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1197490333115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34661795 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15276055 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73413164068864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78008142155776 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74656005179392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79209979527168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1201820332032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚215 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159527424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34671773 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77888708271547 % _timestamp‚1.7843690214639914e+09 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34688289 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1202301754880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1243265650176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1202389698560 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79247659857920 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1198057654645 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74691316329984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74691513336832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73448075583488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79247659857920 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34678297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚164 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76727707013120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19395023 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78045270159360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 trainer/global_step‚2049 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34678297 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14048140 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4640634 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76727689973760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15283274 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1243437753344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34688289 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15283276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006848 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1198143791843 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19405013 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15283274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14047923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceõ€àStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15283276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4640848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 % _timestamp‚1.7843690218092332e+09 _runtime‚ 963.253303855 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19405013 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1202389698560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 âåÊ&­ ÄM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34688289 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1202301754880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1243265650176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1202389698560 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79247659857920 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1198057654645 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74691316329984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74691513336832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73448075583488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79247659857920 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34678297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚164 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76727707013120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19395023 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78045270159360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 trainer/global_step‚2049 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34678297 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14048140 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4640634 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76727689973760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15283274 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1243437753344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34688289 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15283276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006848 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1198143791843 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19405013 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15283274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14047923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15283276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4640848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 % _timestamp‚1.7843690218092332e+09 _runtime‚ 963.253303855 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19405013 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1202389698560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128¤íÞ¼¤0® + _wandbruntime‚963  _runtime‚963^+š¯ ÿ /loss/FRt2/detection_loss‚1.3655511564362592 1loss/SRqqt2/detection_loss‚0.6662337008765653 /loss/FBt1/detection_loss‚1.2481998461816046 /loss/FBt2/detection_loss‚0.5274078501388431 0loss/FRt1/assignment_loss‚2.2474113377657803 /loss/FRt2/assignment_loss‚1.221123283829432 1loss/SRqqt1/assignment_loss‚0.921560713539382 2loss/SRqqt2/assignment_loss‚0.6506697364007491 1loss/FBt1/assignment_loss‚0.03724208396549026 &loss/total_loss‚10.786846510686345 trainer/global_step‚2049 _runtime‚ 963.254906075 1loss/FBt2/assignment_loss‚0.05054409219212841 /loss/FRt1/detection_loss‚0.6570477632119668 1loss/SRqqt1/detection_loss‚1.1938549461481454 _step‚165 epoch‚2 % _timestamp‚1.7843690218110027e+09’8R ° „ epoch‚2 % _timestamp‚1.7843690218110027e+09 /loss/FRt2/detection_loss‚1.3655511564362592 1loss/SRqqt2/detection_loss‚0.6662337008765653 /loss/FBt1/detection_loss‚1.2481998461816046 /loss/FBt2/detection_loss‚0.5274078501388431 0loss/FRt1/assignment_loss‚2.2474113377657803 /loss/FRt2/assignment_loss‚1.221123283829432 1loss/SRqqt1/assignment_loss‚0.921560713539382 2loss/SRqqt2/assignment_loss‚0.6506697364007491 1loss/FBt1/assignment_loss‚0.03724208396549026 &loss/total_loss‚10.786846510686345 trainer/global_step‚2049 _runtime‚ 963.254906075 1loss/FBt2/assignment_loss‚0.05054409219212841 /loss/FRt1/detection_loss‚0.6570477632119668 1loss/SRqqt1/detection_loss‚1.1938549461481454 _step‚165¥o^ ›± ‚(: ‰Ÿíҥ酎 gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚182.63' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.525999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉŽ2ÓÀ² ‚(:µ ‰ŸíÒŽóÍ‘ proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 23.80078125) proc.memory.percent‚16.22464656829834 disk.sda.in‚0 network.sent‚ 2.53583e+06) disk./.usagePercent‚39.60832393546742 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1224.29296875% disk.nvme0n1.out‚2353.34423828125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 23.80078125" memory_percent‚97.800612449646 disk.rbd1.in‚0' disk.nvme0c0n1.out‚2353.34423828125 cpu‚26.660701332912506 disk.sda2.out‚0 disk.dm-1.out‚ 23.80078125 network.recv‚350140$ disk./.usageGB‚590.1926498413086# proc.memory.rssMB‚ 2658.24609375 disk.rbd1.out‚2.375# disk.nvme0c0n1.in‚ 1224.29296875 disk.sda2.in‚0 disk.dm-0.in‚ 1224.29296875 disk.sda3.in‚0( proc.memory.availableMB‚ 360.34765625 disk.sda1.out‚0" disk.dm-0.out‚2353.34423828125-È›œ0³ + _wandbruntime‚985  _runtime‚985Ã-ùÃÅ´ ¿ % _timestamp‚1.7843690440115016e+09 _runtime‚ 985.45526806 _step‚166 & lr-AdamW/pg1‚0.0008457742338985767 & lr-AdamW/pg2‚0.0008457742338985767 trainer/global_step‚2099Á¼Cʵ Ä % _timestamp‚1.7843690440115016e+09 _runtime‚ 985.45526806 _step‚166 & lr-AdamW/pg1‚0.0008457742338985767 & lr-AdamW/pg2‚0.0008457742338985767 trainer/global_step‚2099¦èw‹0¶ + _wandbruntime‚985  _runtime‚985QWm]º'· ´O ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1230320557056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19861620 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78514836744192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9627008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19851644 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199111168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81093922987520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4747813 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚79740708017655 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81094027963392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15637103 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76431058079232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35488747 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79863707406336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14374818 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174207488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79863707406336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75158739800064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15637105 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4748045 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272144071680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 $ _timestamp‚1.784369044012479e+09 _step‚167 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79863690366976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35498725 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19851644 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1225888312823 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1230320557056 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19861620 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15637105 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81093922987520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4747813 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚79740708017655 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81094027963392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15637103 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76431058079232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35488747 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79863707406336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14374818 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174207488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79863707406336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75158739800064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15637105 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4748045 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272144071680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 $ _timestamp‚1.784369044012479e+09 _step‚167 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79863690366976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35498725 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19851644 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1225888312823 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1230320557056 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19861620 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚Ñ–ëM 15637105 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35515241 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75193626411008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9631489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14381686 % _timestamp‚1.7843690443642316e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79900801331200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1230889923584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1230801979904 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1226541771551 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15644326 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1230801979904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9631492 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19860925 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79777850551071 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1226455634353 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81131708294144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79900818370560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4750194 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1272907762176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78551308779520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4749962 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1272735659008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15644324 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76466362070016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75193651314688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81131708294144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 trainer/global_step‚2099 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35505249 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81131603311104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19870915 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35515241 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35505249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76466559076864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15644326 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19870915 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14381451 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15644324 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78551291740160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192íÇa=Ê&» ÄM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9631492 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19860925 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79777850551071 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1226455634353 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81131708294144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79900818370560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4750194 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1272907762176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78551308779520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4749962 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1272735659008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15644324 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76466362070016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75193651314688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81131708294144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172103168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 trainer/global_step‚2099 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35505249 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81131603311104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19870915 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35515241 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35505249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76466559076864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15644326 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19870915 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14381451 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15644324 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78551291740160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _runtime‚ 985.808360545 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35515241 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75193626411008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9631489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14381686 % _timestamp‚1.7843690443642316e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79900801331200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1230889923584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1230801979904 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1226541771551 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15644326 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1230801979904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚168¨ý©¯¤0¼ + _wandbruntime‚985  _runtime‚985´ù½  2loss/SRqqt2/assignment_loss‚0.5663214938076789 1loss/FBt1/assignment_loss‚0.22208703816941125 &loss/total_loss‚10.368309849566959 % _timestamp‚1.7843690443662949e+09 _step‚169 0loss/FRt1/assignment_loss‚2.2191061599132342 2loss/SRqqt1/assignment_loss‚0.8166858824746063 1loss/FBt2/assignment_loss‚0.08046108208376428 1loss/SRqqt1/detection_loss‚1.2698619281278645 1loss/SRqqt2/detection_loss‚0.7769172023534564 epoch‚2 trainer/global_step‚2099 _runtime‚ 985.809877851 /loss/FRt2/detection_loss‚1.3368995902306995 /loss/FBt1/detection_loss‚0.7807250820100308 0loss/FBt2/detection_loss‚0.40680301726496876 /loss/FRt2/assignment_loss‚1.255960753155087 /loss/FRt1/detection_loss‚0.6364806199761581¼à ¾ † 2loss/SRqqt2/assignment_loss‚0.5663214938076789 1loss/FBt1/assignment_loss‚0.22208703816941125 &loss/total_loss‚10.368309849566959 % _timestamp‚1.7843690443662949e+09 _step‚169 0loss/FRt1/assignment_loss‚2.2191061599132342 2loss/SRqqt1/assignment_loss‚0.8166858824746063 1loss/FBt2/assignment_loss‚0.08046108208376428 1loss/SRqqt1/detection_loss‚1.2698619281278645 1loss/SRqqt2/detection_loss‚0.7769172023534564 epoch‚2 trainer/global_step‚2099 _runtime‚ 985.809877851 /loss/FRt2/detection_loss‚1.3368995902306995 /loss/FBt1/detection_loss‚0.7807250820100308 0loss/FBt2/detection_loss‚0.40680301726496876 /loss/FRt2/assignment_loss‚1.255960753155087 /loss/FRt1/detection_loss‚0.6364806199761581©ÍüCM‘¿ ‚(:† ˜ŸíÒ·Œ”‹ gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚188.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*P»N¶À ‚(:« ˜ŸíÒêùÈŒ disk.nvme0n1.out‚ 2359.078125 disk.dm-0.in‚ 1224.29296875 disk.sda.out‚ 24.11328125 disk.sda2.in‚0 disk.dm-0.out‚ 2359.078125( proc.memory.availableMB‚ 360.09765625 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 24.11328125) proc.memory.percent‚16.22464656829834* disk./.usagePercent‚39.608341599987284 disk.rbd1.out‚2.4375 network.recv‚354867# disk.nvme0c0n1.in‚ 1224.29296875 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.598442e+06 disk.sda3.in‚0" disk.nvme0c0n1.out‚ 2359.078125 disk.sda1.out‚0 cpu‚26.654186333574657$ disk./.usageGB‚590.1929130554199 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 1224.29296875 disk.rbd0.in‚0$ memory_percent‚97.80213832855225# proc.memory.rssMB‚ 2658.24609375 disk.sda3.out‚ 24.11328125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0é¢åÿœÁ ‚(:‘ §ŸíÒ·°­Š gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚147.776' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.555200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÝö½ ‚(:² §ŸíÒþ¹šŒ disk.rbd1.out‚2.4375# proc.memory.rssMB‚ 2658.24609375 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.sent‚ 2.661043e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1296.56640625 disk.sda.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1296.56640625) disk./.usagePercent‚39.68253847158341$ disk.nvme0n1.out‚2403.1162109375) proc.memory.percent‚16.22464656829834 cpu‚26.647850760286655 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2403.1162109375! disk.nvme0n1.in‚ 1296.56640625 disk.dm-1.out‚ 24.2578125! disk.dm-0.out‚2403.1162109375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚ 24.2578125 network.recv‚360152 disk.sda3.out‚ 24.2578125# memory_percent‚97.8018045425415( proc.memory.availableMB‚ 360.15234375$ disk./.usageGB‚591.2985000610352 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0+]´P2à - _wandbruntime‚1007  _runtime‚1007S´ÄÉÆÄ À & lr-AdamW/pg1‚0.0008455948609033048 & lr-AdamW/pg2‚0.0008455948609033048 trainer/global_step‚2149 $ _timestamp‚1.784369066395502e+09 _runtime‚1007.839341285 _step‚170z !2ËÅ Å & lr-AdamW/pg1‚0.0008455948609033048 & lr-AdamW/pg2‚0.0008455948609033048 trainer/global_step‚2149 $ _timestamp‚1.784369066395502e+09 _runtime‚1007.839341285 _step‚170ª·Ä2Æ - _wandbruntime‚1007  _runtime‚1007Õ(G’òÇ µO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81719255617536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76904290627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81719255617536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15998155 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4857435 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76904315531264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15998153 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82977971423744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1258820782080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81592810940209 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82978076399616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _runtime‚ 1007.84076364 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80338438510592 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78205924066816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36325677 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843690663967314e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82977971423744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15998155 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1258732845568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14708643 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80338421471232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81719238578176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B¶ŠÉDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1254286292531 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81719238578176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20317546 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1258820782080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9851208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81592707763763 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14708406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4857201 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20317546 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9851205 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1301633439232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82978076399616 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15998153 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1254372429617 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20327522 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1301811841024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1258732845568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14708643 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80338421471232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81719238578176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1254286292531 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81719238578176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20317546 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1258820782080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9851208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81592707763763 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14708406 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4857201 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20317546 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9851205 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1301633439232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82978076399616 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15998153 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1254372429617 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20327522 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1301811841024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82977971423744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1258820782080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81592810940209 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82978076399616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _runtime‚ 1007.84076364 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80338438510592 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78205924066816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36325677 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843690663967314e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82977971423744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192«â²Ø2É - _wandbruntime‚1008  _runtime‚1008ÖÒµ¹Å&Ê ¿M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36342193 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83015651747328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1008.184275298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1259302204928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1259390148608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81629747120621 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76939202142208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20326827 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81756349542400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16005376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4859364 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80374910545920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16005376 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14715053 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9855692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36332201 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81756349542400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16005374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20326827 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784369066740211e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16005374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1302393656320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1302225747456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83015756730368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78241427889664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚172 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36342193 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20336817 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76939227045888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36332201 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78241620702208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80374893506560 trainer/global_step‚2149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1259390148608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4859589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20336817 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14715281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1254853614061 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1254939751259 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81756366581760 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¬Èþå]Ë ÄM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784369066740211e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16005374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1302393656320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1302225747456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83015756730368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78241427889664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚172 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36342193 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20336817 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812544 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76939227045888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36332201 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78241620702208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80374893506560 trainer/global_step‚2149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1259390148608 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4859589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20336817 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14715281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1254853614061 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1254939751259 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81756366581760 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36342193 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83015651747328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1008.184275298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1259302204928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1259390148608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81629747120621 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76939202142208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20326827 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81756349542400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16005376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 QEDeviceStatsMonitor.on_train_batch_end§O&Am/inactive_split.small_pool.freed‚4859364 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80374910545920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16005376 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14715053 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9855692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36332201 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81756349542400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16005374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20326827¬~œ2Ì - _wandbruntime‚1008  _runtime‚1008¤àD›Í € _step‚173 2loss/SRqqt1/assignment_loss‚0.6958547548536667 /loss/FBt1/detection_loss‚0.7952323768820081 &loss/total_loss‚10.186950108680922 trainer/global_step‚2149 $ _timestamp‚1.784369066741959e+09 1loss/FBt1/assignment_loss‚0.04932140637122627 2loss/FBt2/assignment_loss‚0.031234290283648204 2loss/SRqqt2/assignment_loss‚0.6441621532307507 -loss/FRt2/detection_loss‚1.38370240608166 epoch‚2 _runtime‚1008.185503337 0loss/FRt1/assignment_loss‚2.2672041612185114 0loss/FRt2/assignment_loss‚1.3677033919027481 /loss/FRt1/detection_loss‚0.6282415493955105 1loss/SRqqt1/detection_loss‚1.0541920586425158 1loss/SRqqt2/detection_loss‚0.7236835593816393 /loss/FBt2/detection_loss‚0.5464180004370364Lºb Î … 2loss/SRqqt2/assignment_loss‚0.6441621532307507 -loss/FRt2/detection_loss‚1.38370240608166 epoch‚2 _runtime‚1008.185503337 0loss/FRt1/assignment_loss‚2.2672041612185114 0loss/FRt2/assignment_loss‚1.3677033919027481 /loss/FRt1/detection_loss‚0.6282415493955105 1loss/SRqqt1/detection_loss‚1.0541920586425158 1loss/SRqqt2/detection_loss‚0.7236835593816393 /loss/FBt2/detection_loss‚0.5464180004370364 _step‚173 2loss/SRqqt1/assignment_loss‚0.6958547548536667 /loss/FBt1/detection_loss‚0.7952323768820081 &loss/total_loss‚10.186950108680922 trainer/global_step‚2149 $ _timestamp‚1.784369066741959e+09 1loss/FBt1/assignment_loss‚0.04932140637122627 2loss/FBt2/assignment_loss‚0.031234290283648204­$V|ˆ‘Ï ‚(:† ¶ŸíÒª²’‹ gpu.0.gpu‚60.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÔ ÑÙÐ ‚(:Î ¶ŸíÒˆ÷ÌŒ cpu‚26.64213356936333 proc.cpu.threads‚17 network.sent‚ 2.66606e+06$ disk./.usageGB‚591.2607574462891 disk.sda2.out‚0# proc.memory.rssMB‚ 2658.24609375 disk.sda1.in‚0" disk.dm-0.out‚2410.91650390625 disk.drbd2165.in‚0 network.recv‚363315 disk.dm-1.out‚ 25.45703125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 1296.56640625 disk.sda3.out‚ 25.45703125! disk.nvme0n1.in‚ 1296.56640625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1296.56640625$ memory_percent‚97.80211448669434' proc.memory.availableMB‚ 360.1015625' disk.nvme0c0n1.out‚2410.91650390625% disk.nvme0n1.out‚2410.91650390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 disk.sda.in‚ 0.10546875 disk.rbd1.in‚0) disk./.usagePercent‚39.68000553303963 disk.rbd0.out‚0 disk.sda.out‚ 25.45703125 disk.rbd1.out‚2.5) proc.memory.percent‚16.22464656829834!;Ù"2Ñ - _wandbruntime‚1030  _runtime‚1030SÇÒ Á _runtime‚1030.484342996 _step‚174 & lr-AdamW/pg1‚0.0008454087356968661 & lr-AdamW/pg2‚0.0008454087356968661 trainer/global_step‚2199 % _timestamp‚1.7843690890405416e+09MoeIÌÓ Æ % _timestamp‚1.7843690890405416e+09 _runtime‚1030.484342996 _step‚174 & lr-AdamW/pg1‚0.0008454087356968661 & lr-AdamW/pg2‚0.0008454087356968661 trainer/global_step‚2199®½¢rý2Ô - _wandbruntime‚1030  _runtime‚1030Êï—ç¾'Õ ¸O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1287321007104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10075405 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1282684272239 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83444707509871 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37142651 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84862019859968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83574786789376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16359205 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16359203 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1331324064768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20793424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78649866358784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1287233070592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16359205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82162023237632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84862019859968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84862124835840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20783448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4965810 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15041218 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78649891262464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚1030.485666566 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20783448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20793424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84862124835840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82162040276992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1287321007104 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79981012021760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83574803828736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10075408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4965571 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15040976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1287233070592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 % _timestamp‚1.7843690890416775e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16359203 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83444810686317 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139²Ýù¯Ã'Ö ½O YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15040976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1287233070592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 % _timestamp‚1.7843690890416775e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16359203 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83444810686317 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1287321007104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10075405 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1282684272239 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83444707509871 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37142651 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84862019859968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83574786789376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16359205 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16359203 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1331324064768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20793424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78649866358784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1287233070592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16359205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82162023237632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84862019859968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84862124835840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20783448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4965810 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15041218 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78649891262464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚1030.485666566 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20783448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20793424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84862124835840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82162040276992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1287321007104 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79981012021760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83574803828736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10075408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4965571 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0¯ÉÕë·2× - _wandbruntime‚1030  _runtime‚1030rÞÈÚØ ÂM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84899805166592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1331911688192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84899805166592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83611897753600 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1287890373632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78684802777088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20802719 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1287802429952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37169145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15047623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37169145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10079889 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82198495272960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16366426 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80016714465280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83611914792960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37159153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1283251593769 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1287890373632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20792729 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20802719 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83481746866729 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 REDeviceStatsMonitomÒ!eî r.on_train_batch_end/allocation.large_pool.allocated‚16366426 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15047854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4967962 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82198512312320 % _timestamp‚1.7843690893827069e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4967734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37159153 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16366424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1331733293568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78684777873408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84899700183552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83481850043287ËÒÁÍ&Ù ÇM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78684802777088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20802719 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1287802429952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37169145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15047623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37169145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10079889 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82198495272960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16366426 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80016714465280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83611914792960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37159153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1283251593769 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1287890373632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20792729 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20802719 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83481746866729 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16366426 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15047854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4967962 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82198512312320 % _timestamp‚1.7843690893827069e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4967734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37159153 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16366424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1331733293568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78684777873408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84899700183552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83481850043287 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84899805166592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1331911688192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84899805166592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83611897753600 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1287890373632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2°K!§É2Ú - _wandbruntime‚1030  _runtime‚1030°]èÛ þ /loss/FRt1/assignment_loss‚2.257349453751351 .loss/FRt1/detection_loss‚0.629641040684733 /loss/FBt2/detection_loss‚0.4925006396864375 2loss/SRqqt1/assignment_loss‚0.7627226590105382 1loss/FBt1/assignment_loss‚0.41535937814181745 0loss/FBt2/assignment_loss‚0.1531280893246772 0loss/SRqqt1/detection_loss‚1.106628744127719 1loss/SRqqt2/detection_loss‚0.6723312883852766 epoch‚2 _step‚177 2loss/SRqqt2/assignment_loss‚0.5698372914552098 /loss/FBt1/detection_loss‚1.2948599696943635 &loss/total_loss‚10.703544249525148 trainer/global_step‚2199 _runtime‚1030.827528508 0loss/FRt2/assignment_loss‚1.1727290093898772 /loss/FRt2/detection_loss‚1.1764566858731476 % _timestamp‚1.7843690893839798e+09FqaŠ Ü ƒ 1loss/FBt1/assignment_loss‚0.41535937814181745 0loss/FBt2/assignment_loss‚0.1531280893246772 0loss/SRqqt1/detection_loss‚1.106628744127719 1loss/SRqqt2/detection_loss‚0.6723312883852766 epoch‚2 _step‚177 2loss/SRqqt2/assignment_loss‚0.5698372914552098 /loss/FBt1/detection_loss‚1.2948599696943635 &loss/total_loss‚10.703544249525148 trainer/global_step‚2199 _runtime‚1030.827528508 0loss/FRt2/assignment_loss‚1.1727290093898772 /loss/FRt2/detection_loss‚1.1764566858731476 % _timestamp‚1.7843690893839798e+09 /loss/FRt1/assignment_loss‚2.257349453751351 .loss/FRt1/detection_loss‚0.629641040684733 /loss/FBt2/detection_loss‚0.4925006396864375 2loss/SRqqt1/assignment_loss‚0.7627226590105382±B‰Í^‘Ý ‚(:† ÅŸíÒšŠ‹ gpu.0.gpu‚61.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚179.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#7×ÍÞ ‚(: şíÒ¤‚€ disk.sda2.in‚0$ disk./.usageGB‚590.1942939758301 disk.rbd0.in‚0 disk.sda.out‚ 25.671875 disk.sda1.out‚0" disk.dm-0.out‚3540.20556640625 disk.rbd1.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1296.56640625 disk.dm-1.out‚ 25.671875 proc.cpu.threads‚17 network.recv‚368210! disk.nvme0n1.in‚ 1296.56640625% disk.nvme0n1.out‚3540.20556640625$ memory_percent‚98.19626808166504 proc.memory.rssMB‚2658.25) proc.memory.percent‚16.22467041015625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 disk.drbd2165.out‚0' proc.memory.availableMB‚ 295.5234375 disk.sda2.out‚0 network.sent‚ 2.72862e+06' disk.nvme0c0n1.out‚3540.20556640625 disk.dm-0.in‚ 1296.56640625 disk.sda3.out‚ 25.671875 disk.drbd2165.in‚0) disk./.usagePercent‚39.60843427471468 disk.sda.in‚ 0.10546875 disk.rbd1.out‚2.5 cpu‚26.63587196536244ž¶û‘ß ‚(:† ÔŸíÒŽ±Š gpu.0.gpu‚29.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09uˆRÕà ‚(:Ê ÔŸíҮͦŒ disk.rbd1.in‚0 disk.sda3.in‚ 0.10546875# proc.memory.availableMB‚295.375 network.sent‚ 2.791412e+06 network.recv‚373102! disk.nvme0n1.in‚ 1296.56640625 disk.rbd0.in‚0 disk.dm-0.in‚ 1296.56640625 disk.sda.out‚ 26.20703125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚3546.07958984375 disk.sda2.out‚0 disk.sda3.out‚ 26.20703125$ memory_percent‚98.19717407226562 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.10546875 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚39.608434530722214$ disk./.usageGB‚590.1942977905273% disk.nvme0n1.out‚3546.07958984375 disk.dm-1.in‚ 0.10546875 proc.memory.rssMB‚2658.25 disk.sda2.in‚0" disk.dm-0.out‚3546.07958984375) proc.memory.percent‚16.22467041015625 cpu‚26.630304974157205 disk.rbd1.out‚2.5625# disk.nvme0c0n1.in‚ 1296.56640625 disk.dm-1.out‚ 26.20703125ìÏÔ2á - _wandbruntime‚1052  _runtime‚1052|LPÆâ À trainer/global_step‚2249 $ _timestamp‚1.784369111512258e+09 _runtime‚1052.956050404 _step‚178 & lr-AdamW/pg1‚0.0008452158612604338 & lr-AdamW/pg2‚0.0008452158612604338nÎËã Å _runtime‚1052.956050404 _step‚178 & lr-AdamW/pg1‚0.0008452158612604338 & lr-AdamW/pg2‚0.0008452158612604338 trainer/global_step‚2249 $ _timestamp‚1.784369111512258e+09²›ud 2ä - _wandbruntime‚1052  _runtime‚1052NξX¾'å ¸O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85430352039936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85430352039936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81756035296768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1360771608576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1360593206784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15375187 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80395466993664 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1315733295616 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86746173272064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21249350 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15375429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5075582 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16720253 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85430335000576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311082251947 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86746173272064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10299608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚1052.957267644 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16720255 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85296707255979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37969603 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86746068296192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10299605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5075821 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315821232128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16720255 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83985642043392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21259326 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83985625004032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315821232128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86746068296192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1315733295616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81756238602240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21259326 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85430335000576 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85296810432425 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚179ª|2&æ ½O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1360771608576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1360593206784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15375187 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80395466993664 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1315733295616 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86746173272064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21249350 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15375429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5075582 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16720253 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85430335000576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311082251947 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86746173272064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10299608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚1052.957267644 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16720255 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85296707255979 VGDeviceStatsMonitor.on_train_batch_sc«w¯tart/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37969603 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86746068296192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10299605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5075821 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315821232128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16720255 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83985642043392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21259326 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83985625004032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315821232128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86746068296192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1315733295616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81756238602240 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21259326 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85430335000576 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85296810432425 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚179 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85430352039936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85430352039936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81756035296768³? S2ç - _wandbruntime‚1053  _runtime‚1053$åUÆÆ&è ÀM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10304089 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5077917 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 % _timestamp‚1.7843691118550887e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1316302654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37986105 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16727474 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86783853602816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80430378508288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81791735065600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1361356557312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86783853602816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1361180259840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84022097039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85467463004160 trainer/global_step‚2249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15382009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80430353604608 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85333849789395 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 1053.2991957 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21258631 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85467445964800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5077698 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37996097 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16727474 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21268621 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86783748619776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85467445964800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15381787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1311735710675 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1316390598656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473CtË&é ÅM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86783748619776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85467445964800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15381787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1311735710675 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1316390598656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10304089 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5077917 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 % _timestamp‚1.7843691118550887e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1316302654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37986105 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16727474 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86783853602816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80430378508288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81791735065600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1361356557312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86783853602816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1361180259840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84022097039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85467463004160 trainer/global_step‚2249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15382009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80430353604608 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85333849789395 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 1053.2991957 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21258631 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85467445964800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5077698 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37996097 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16727474 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21268621 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3´½ýÑ-2ê - _wandbruntime‚1053  _runtime‚1053Jdéë þ 2loss/SRqqt2/assignment_loss‚0.6008043225450416 1loss/FBt1/assignment_loss‚0.06731187938393782 /loss/FRt2/detection_loss‚1.2490371927537534 1loss/SRqqt2/detection_loss‚0.7193941177554533 /loss/FBt2/detection_loss‚0.5148844309151173 $loss/total_loss‚9.63444199914376 epoch‚2 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚2.0773157024083018 0loss/FRt2/assignment_loss‚1.2720982217063468 1loss/FBt2/assignment_loss‚0.07321221606756702 _step‚181 /loss/SRqqt1/detection_loss‚1.06532346829772 2loss/SRqqt1/assignment_loss‚0.7292379741212008 /loss/FRt1/detection_loss‚0.6201685265980454 /loss/FBt1/detection_loss‚0.6456539465912751 % _timestamp‚1.7843691118567343e+09 _runtime‚1053.300278045pR«i ì ƒ /loss/SRqqt1/detection_loss‚1.06532346829772 2loss/SRqqt1/assignment_loss‚0.7292379741212008 /loss/FRt1/detection_loss‚0.6201685265980454 /loss/FBt1/detection_loss‚0.6456539465912751 % _timestamp‚1.7843691118567343e+09 _runtime‚1053.300278045 2loss/SRqqt2/assignment_loss‚0.6008043225450416 1loss/FBt1/assignment_loss‚0.06731187938393782 /loss/FRt2/detection_loss‚1.2490371927537534 1loss/SRqqt2/detection_loss‚0.7193941177554533 /loss/FBt2/detection_loss‚0.5148844309151173 $loss/total_loss‚9.63444199914376 epoch‚2 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚2.0773157024083018 0loss/FRt2/assignment_loss‚1.2720982217063468 1loss/FBt2/assignment_loss‚0.07321221606756702 _step‚181µj67®œí ‚(:‘ ãŸíÒÌÇðŠ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CžIÑî ‚(:Æ ãŸíÒëÕöŒ cpu‚26.62482352387946 disk.dm-1.out‚ 26.4765625 disk.dm-0.in‚ 1361.64453125 disk.sda.out‚ 26.4765625# memory_percent‚98.1978178024292 disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875& disk.nvme0c0n1.out‚3549.7646484375 disk.sda1.in‚0 disk.rbd1.out‚2.5625# disk.nvme0c0n1.in‚ 1361.64453125( proc.memory.availableMB‚ 295.26953125$ disk./.usageGB‚590.1562995910645 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 26.4765625 disk.sda.in‚ 0.10546875 disk.drbd2165.out‚0) proc.memory.percent‚16.22467041015625 proc.cpu.threads‚17 disk.rbd1.in‚0 network.sent‚ 2.796354e+06) disk./.usagePercent‚39.60588443967365! disk.nvme0n1.in‚ 1361.64453125! disk.dm-0.out‚3549.7646484375 disk.drbd2165.in‚0 disk.sda1.out‚0 network.recv‚376390$ disk.nvme0n1.out‚3549.7646484375 disk.sda2.in‚0 proc.memory.rssMB‚2658.25>?ÈI2ï - _wandbruntime‚1075  _runtime‚1075i•ÒÇð Á & lr-AdamW/pg1‚0.0008450162406832834 & lr-AdamW/pg2‚0.0008450162406832834 trainer/global_step‚2299 % _timestamp‚1.7843691341124446e+09 _runtime‚1075.556233584 _step‚182l“SÌñ Æ _step‚182 & lr-AdamW/pg1‚0.0008450162406832834 & lr-AdamW/pg2‚0.0008450162406832834 trainer/global_step‚2299 % _timestamp‚1.7843691341124446e+09 _runtime‚1075.556233584¶Á;¯2ò - _wandbruntime‚1075  _runtime‚1075ìb¸Ó› ó ¶O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87285883211776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1344233520640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10523805 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843691341134543e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5185053 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚2299 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38806533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85809243809792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38796555 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15709099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.onéÊt!_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17081305 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17081305 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10523808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21715252 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88630116732416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1339566368741 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1390100657152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88630116732416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87285900251136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88630221708288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21725228 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15708858 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1339480231655 _step‚183 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5185291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83531118478336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976!Ùl¥Á'ô »O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17081303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82141017821184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87148810178533 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1390279058944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1344321457152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85809226770432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87285900251136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10523805 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843691341134543e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5185053 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚2299 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38806533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85809243809792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38796555 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15709099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17081305 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17081305 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10523808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104975872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21715252 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572139 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88630116732416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87936512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1339566368741 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1390100657152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88630116732416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87285900251136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88630221708288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21725228 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15708858 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203305472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1339480231655 _step‚183 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5185291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83531118478336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87285883211776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1344233520640 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21734523 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚2299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1344802880000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87185849535503 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87323011215360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17088526 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87323011215360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83566614514688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1344890823680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1344802880000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88667902039040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5187528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1390685178880 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15715578 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38823049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21724533 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88667902039040 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1390867767808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1340133690383 _runtime‚1075.900716511 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38813057 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21724533 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85845715845120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38823049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17088524 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17088524 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87185746358945 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10528289 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82175929335808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1344890823680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15715820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82175954239488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1340047553185 % _timestamp‚1.7843691344566483e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17088526 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21734523 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚2299 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1344802880000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87185849535503 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87323011215360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17088526 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87323011215360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83566614514688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1344890823680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1344802880000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88667902039040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5187528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1390685178880 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15715578 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38823049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21724533 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88667902039040 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300375552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1390867767808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1340133690383 _runtime‚1075.900716511 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38813057 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceœvXÌStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21724533 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85845715845120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38823049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17088524 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17088524 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87185746358945 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10528289 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82175929335808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1344890823680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15715820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82175954239488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1340047553185 % _timestamp‚1.7843691344566483e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17088526 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87632845576192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚89000706748195 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 % _timestamp‚1.7843691565036309e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10748008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90514165168640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5295066 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90514270144512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _step‚187 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1372733745664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16043309 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90514270144512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5295301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22181154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1372821682176 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90514165168640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5295066 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90514270144512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _step‚187 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1372733745664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16043309 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90514270144512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5295301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22181154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137086 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1372821682176 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87632845576192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚89000706748195 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 % _timestamp‚1.7843691565036309e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081669632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10748008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39650001 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1368531670091 $ _timestamp‚1.784369156856594e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17449576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22200425 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1373391048704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83921529970688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16049999 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89178559426560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83921505067008 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39640009 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5297507 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22200425 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85341870772224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚90551950475264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16049779 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89037849281611 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39650001 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚2349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17449576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5297507 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22200425 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85341870772224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚90551950475264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16049779 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89037849281611 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39650001 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚2349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17449576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39650001 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1368531670091 $ _timestamp‚1.784369156856594e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17449576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22200425 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1373391048704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83921529970688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16049999 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327¼U¾u2ˆ - _wandbruntime‚1098  _runtime‚1098_ã§Þ‰ ÿ 1loss/SRqqt1/detection_loss‚1.1549411031898835 /loss/FBt1/detection_loss‚1.2143795974552631 _runtime‚1098.301897977 0loss/SRqqt1/assignment_loss‚0.82881445275704 2loss/FBt2/assignment_loss‚0.008262477156135414 epoch‚2 /loss/FRt1/detection_loss‚0.6698319628680403 1loss/SRqqt2/detection_loss‚0.7839727642460608 &loss/total_loss‚10.567128590652953 trainer/global_step‚2349 % _timestamp‚1.7843691568579528e+09 0loss/FRt2/assignment_loss‚1.2220996256624714 2loss/SRqqt2/assignment_loss‚0.6990190051659995 /loss/FBt1/assignment_loss‚0.055602406293076 0loss/FBt2/detection_loss‚0.44236490045275006 _step‚189 0loss/FRt1/assignment_loss‚2.1278390635433513 .loss/FRt2/detection_loss‚1.360001231862881^Œr Š „ 0loss/SRqqt1/assignment_loss‚0.82881445275704 2loss/FBt2/assignment_loss‚0.008262477156135414 epoch‚2 /loss/FRt1/detection_loss‚0.6698319628680403 1loss/SRqqt2/detection_loss‚0.7839727642460608 &loss/total_loss‚10.567128590652953 trainer/global_step‚2349 % _timestamp‚1.7843691568579528e+09 0loss/FRt2/assignment_loss‚1.2220996256624714 2loss/SRqqt2/assignment_loss‚0.6990190051659995 /loss/FBt1/assignment_loss‚0.055602406293076 0loss/FBt2/detection_loss‚0.44236490045275006 _step‚189 0loss/FRt1/assignment_loss‚2.1278390635433513 .loss/FRt2/detection_loss‚1.360001231862881 1loss/SRqqt1/detection_loss‚1.1549411031898835 /loss/FBt1/detection_loss‚1.2143795974552631 _runtime‚1098.301897977½pû®y‘‹ ‚(:†  íÒœ‘Œ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚152.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢©=ÔŒ ‚(:É  íÒØèÒŒ disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1361.64453125 proc.cpu.threads‚17" disk.dm-0.out‚3565.08154296875 disk.sda1.in‚0 disk.sda3.out‚ 27.43359375$ disk./.usageGB‚590.1561622619629 disk.rbd0.out‚0* disk./.usagePercent‚39.605875223402414' disk.nvme0c0n1.out‚3565.08154296875# memory_percent‚98.1968879699707 disk.drbd2165.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚3565.08154296875& proc.memory.availableMB‚ 295.421875 network.recv‚388811 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 27.43359375 disk.sda.in‚ 0.10546875 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 1361.64453125 disk.sda3.in‚ 0.10546875 disk.sda.out‚ 27.43359375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1361.64453125 proc.memory.rssMB‚2658.25) proc.memory.percent‚16.22467041015625 network.sent‚ 2.925751e+06 disk.sda1.out‚0 disk.rbd1.out‚2.75 cpu‚26.60891266688643¶Îç’2 - _wandbruntime‚1120  _runtime‚1120‡»óPÇŽ Á trainer/global_step‚2399 % _timestamp‚1.7843691789806826e+09 _runtime‚1120.425626691 _step‚190 & lr-AdamW/pg1‚0.0008445967740041453 & lr-AdamW/pg2‚0.0008445967740041453å¸ö:Ì Æ _step‚190 & lr-AdamW/pg1‚0.0008445967740041453 & lr-AdamW/pg2‚0.0008445967740041453 trainer/global_step‚2399 % _timestamp‚1.7843691789806826e+09 _runtime‚1120.425626691¾Iæt2 - _wandbruntime‚1120  _runtime‚1120Y²ܾ'‘ ¸O G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17803405 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85632169283584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1401321907200 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40460437 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22657032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22647056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1449183970304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22657032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1401233970688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90852706494303 _runtime‚1120.425718955 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90996996673536 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87081181143552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90996979634176 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92398213604864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92398213604864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90996996673536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843691789814222e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40450459 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1449011859968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22647056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90852809670749 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1401321907200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16376958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10972208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5404750 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17803403 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10972205 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90996979634176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16376733 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92398213604864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90996996673536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843691789814222e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40450459 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300375552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1449011859968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22647056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90852809670749 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300375552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1401321907200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QDDevicCÂÿÞ eStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16376958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10972208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5404750 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668603 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17803403 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10972205 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87936512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90996979634176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16376733 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1401321907200 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081669632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40460437 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22657032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193107968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22647056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1449183970304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22657032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1401233970688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90852706494303 _runtime‚1120.425718955 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104975872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90996996673536 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87081181143552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¿=‘>2“ - _wandbruntime‚1120  _runtime‚11202ã!(È&” ÂM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90889849027719 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16383551 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16383316 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137198 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40466961 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87943680 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚2399 % _timestamp‚1.7843691793218586e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17810624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1401891273728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85667105701888 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92435998911488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10976689 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40476953 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85667080798208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449773703168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22666327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40466961 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1396843512601 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17810626 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1401803330048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89492902338560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91034090598400 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92435893928448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17810626 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90889745851161 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22656337 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17810624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5406627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1401803330048 _runtime‚1120.765876507 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92435998911488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40476953 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22666327 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91034107637760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91034107637760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91034090598400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22656337 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10976692 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5406859 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1396929649799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87116678203904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1449597405696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚2399 % _timestamp‚1.7843691793218586e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17810624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1401891273728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87943680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85667105701888 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92435998911488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10976689 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40476953 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85667080798208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449773703168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22666327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40466961 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1396843512601 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17810626 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1401803330048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89492902338560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91034090598400 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92435893928448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081669632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17810626 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90889745851161 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22656337 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17810624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668603 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5406627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201201152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1401803330048 _runtime‚1120.765876507 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92435998911488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300375552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40476953 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22666327 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91034107637760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91034107637760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91034090598400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176558 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176297472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22656337 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081669632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10976692 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5406859 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1396929649799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87116678203904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1449597405696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16383551 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16383316À¡&á2– - _wandbruntime‚1120  _runtime‚1120OçV— € /loss/FRt1/assignment_loss‚2.170952752362127 2loss/SRqqt2/assignment_loss‚0.5832806682699363 .loss/FRt2/detection_loss‚1.295875739719531 /loss/FBt1/detection_loss‚0.6026001632213592 trainer/global_step‚2399 0loss/FRt2/assignment_loss‚1.2697551054085872 2loss/FBt1/assignment_loss‚0.001178834520252773 2loss/FBt2/assignment_loss‚0.047577803503058215 /loss/FRt1/detection_loss‚0.6294949539684793 epoch‚2 % _timestamp‚1.7843691793236399e+09 1loss/SRqqt1/detection_loss‚1.1814659614617944 $loss/total_loss‚9.62648720568715 2loss/SRqqt1/assignment_loss‚0.8163317342483644 1loss/SRqqt2/detection_loss‚0.6570193432624828 /loss/FBt2/detection_loss‚0.3709541457411769 _runtime‚1120.767180739 _step‚193POÚG ˜ … 2loss/SRqqt1/assignment_loss‚0.8163317342483644 1loss/SRqqt2/detection_loss‚0.6570193432624828 /loss/FBt2/detection_loss‚0.3709541457411769 _runtime‚1120.767180739 _step‚193 /loss/FRt1/assignment_loss‚2.170952752362127 2loss/SRqqt2/assignment_loss‚0.5832806682699363 .loss/FRt2/detection_loss‚1.295875739719531 /loss/FBt1/detection_loss‚0.6026001632213592 trainer/global_step‚2399 0loss/FRt2/assignment_loss‚1.2697551054085872 2loss/FBt1/assignment_loss‚0.001178834520252773 2loss/FBt2/assignment_loss‚0.047577803503058215 /loss/FRt1/detection_loss‚0.6294949539684793 epoch‚2 % _timestamp‚1.7843691793236399e+09 1loss/SRqqt1/detection_loss‚1.1814659614617944 $loss/total_loss‚9.62648720568715ÁÃÞ³€‘™ ‚(:† Ÿ íÒ±¢¡Š gpu.0.gpu‚56.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú†ˆÐš ‚(:Å Ÿ íÒ¸”ÏŒ disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.75! proc.memory.rssMB‚ 2660.890625 network.sent‚ 2.988335e+06 disk.sda3.in‚ 0.10546875% disk.nvme0c0n1.out‚3569.638671875 disk.sda.in‚ 0.10546875( proc.memory.availableMB‚ 292.66015625 disk.dm-1.out‚ 27.6015625 disk.dm-0.in‚ 1433.4765625 disk.drbd2165.in‚0# disk.nvme0n1.out‚3569.638671875 disk.rbd0.in‚0 disk.dm-0.out‚3569.638671875 cpu‚26.61682288008192* disk./.usagePercent‚39.605874711387344" disk.nvme0c0n1.in‚ 1433.4765625 disk.dm-1.in‚ 0.10546875 disk.sda3.out‚ 27.6015625 disk.drbd2165.out‚0$ memory_percent‚98.21374416351318 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 27.6015625* proc.memory.percent‚16.240787506103516 proc.cpu.threads‚17 network.recv‚393945$ disk./.usageGB‚590.1561546325684 disk.nvme0n1.in‚ 1433.4765625 disk.sda2.out‚0´¾ãBœ› ‚(:‘ ® íÒ¶‹ÅŠ gpu.0.gpu‚68.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚28.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý[’ Õœ ‚(:Ê ® íÒ“òìŒ& proc.memory.availableMB‚ 278.140625 network.recv‚398577$ disk./.usageGB‚590.1556701660156 disk.drbd2165.out‚0 network.sent‚ 3.050859e+06* disk./.usagePercent‚39.605842198430494 disk.rbd1.out‚2.8125# proc.memory.rssMB‚ 2675.41796875 disk.nvme0n1.in‚ 1433.4765625 disk.sda2.out‚0 disk.dm-1.out‚ 27.8671875 disk.sda3.in‚ 0.10546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚3574.4443359375 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚3574.4443359375$ disk.nvme0n1.out‚3574.4443359375* proc.memory.percent‚16.329455375671387 cpu‚26.67825696817167 disk.sda3.out‚ 27.8671875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1433.4765625$ memory_percent‚98.30236434936523 proc.cpu.threads‚17 disk.sda.in‚ 0.10546875 disk.sda2.in‚0 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 1433.4765625 disk.sda.out‚ 27.8671875û¬Ÿœ2 - _wandbruntime‚1143  _runtime‚1143ÉÓ©Šž ©{ jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5225472450256348 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6889469027519226 +jet/accuracy_1_of_1‚0.23764654994010925 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996249258518219 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6889469027519226 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.575833797454834 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06968951970338821 7 Purityó7eH%;/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5693259239196777 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893074035644531 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6183201670646667 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06873094290494919 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5080221891403198 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5251498222351074 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9968903660774231 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5720492601394653 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024426616728305817 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26555517315864563 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2152777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06426657736301422 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2654312551021576 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010736656375229359 +jet/accuracy_3_of_3‚0.06244491785764694 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957171082496643 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.618388295173645 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5255205631256104 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06687308102846146 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 'particle/f_score‚0.6041366457939148 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5555555820465088 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06753221899271011 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8827873468399048 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08349957317113876 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (loss/val_total_loss‚9.91148953831267 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07185770571231842 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5761352181434631 ,jet/accuracy_4_of_4‚0.012545786798000336 .particle/accuracy_3_of_3‚0.999514102935791 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚1143.160185272 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16978827118873596 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5723279714584351 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09479166567325592 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886391162872314 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5337367057800293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5931751728057861 *jet/accuracy_1_of_2‚0.6740078926086426 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5741066336631775 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936507344245911 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9914529323577881 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43772584199905396 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_2‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6763157248497009 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13119612634181976 +jet/accuracy_2_of_2‚0.11628363281488419 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5732566714286804 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5195426344871521 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8299999833106995 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950514435768127 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6562386751174927 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.40245479345321655 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927893280982971 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07525590062141418 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006514979992061853 particle/accuracy_2_of_2‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7204449772834778 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0642085149884224 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.051620036363601685 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08426923304796219 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14783577620983124 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9180018305778503 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9884023070335388 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978419542312622 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 trainer/global_step‚2402 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21328812837600708 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994543731212616 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5340465307235718 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.847612738609314 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4050000011920929 % _timestamp‚1.7843692017157178e+09 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901549816131592 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5351678133010864 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6527708768844604 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019566211849451065 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.9027735590934753 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06910452246665955 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40123456716537476 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.532484769821167 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5258440971374512 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7465811371803284 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.626311182975769 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7554638981819153 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6196580529212952 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940857887268066 +jet/accuracy_2_of_3‚0.41075998544692993 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5363295674324036 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5074779987335205 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05337774008512497 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035234359093010426 'particle/accuracy‚0.834902822971344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943981170654297 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08188078552484512 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10058094561100006 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.5945784449577332 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06538239866495132 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9944444298744202 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6419483423233032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07422163337469101 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.118327297270298 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11613143980503082 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.25798818469047546 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5959476828575134 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11207333952188492 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8965535759925842 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9768046140670776 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7523182034492493 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 epoch‚2 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5850760340690613 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7311403155326843 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_4‚0.9387729167938232 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3138899803161621 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08092188090085983 +jet/accuracy_3_of_4‚0.08542021363973618 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970965385437012 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3098651170730591 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5750643014907837 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08328856527805328 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9968903660774231 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10322272032499313 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6756049394607544 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8299999833106995 .particle/accuracy_3_of_4‚0.995192289352417 +particle/sensitivity‚0.5867357850074768 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09006567299365997 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6668245196342468 6validation_average_jet_accuracy‚0.2851008474826813 _step‚194 *jet/accuracy_1_of_3‚0.8620404005050659 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8374501466751099 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6269627809524536 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4535508453845978 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 -particle/accuracy_4_of_4‚0.93010413646698 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08169447630643845 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898526072502136 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5701525807380676 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06695180386304855 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5966700911521912 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07618562132120132 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08310969173908234 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013618266209959984 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4519169330596924 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953316450119019 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05337774008512497 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5958793759346008 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928867816925049 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4544270932674408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5363295674324036 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.538003146648407 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06687308102846146 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6717374324798584 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1"=Ý´=Ÿ ®{ epoch‚2 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5850760340690613 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7311403155326843 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_4‚0.9387729167938232 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3138899803161621 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08092188090085983 +jet/accuracy_3_of_4‚0.08542021363973618 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970965385437012 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3098651170730591 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5750643014907837 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08328856527805328 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9968903660774231 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10322272032499313 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6756049394607544 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8299999833106995 .particle/accuracy_3_of_4‚0.995192289352417 +particle/sensitivity‚0.5867357850074768 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09006567299365997 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6668245196342468 6validation_average_jet_accuracy‚0.2851008474826813 _step‚194 *jet/accuracy_1_of_3‚0.8620404005050659 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8374501466751099 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6269627809524536 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4535508453845978 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 -particle/accuracy_4_of_4‚0.93010413646698 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08169447630643845 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898526072502136 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5701525807380676 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06695180386304855 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5966700911521912 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07618562132120132 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08310969173908234 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013618266209959984 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4519169330596924 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953316450119019 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05337774008512497 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5958793759346008 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928867816925049 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4544270932674408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5363295674324036 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.538003146648407 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06687308102846146 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6717374324798584 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5225472450256348 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6889469027519226 +jet/accuracy_1_of_1‚0.23764654994010925 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996249258518219 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6889469027519226 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.575833797454834 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06968951970338821 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5693259239196777 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893074035644531 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6183201670646667 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06873094290494919 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5080221891403198 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5251498222351074 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9968903660774231 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5720492601394653 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024426616728305817 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26555517315864563 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2152777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06426657736301422 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2654312551021576 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010736656375229359 +jet/accuracy_3_of_3‚0.06244491785764694 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957171082496643 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.618388295173645 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5255205631256104 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06687308102846146 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 'particle/f_score‚0.6041366457939148 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5555555820465088 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06753221899271011 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8827873468399048 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08349957317113876 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (loss/val_total_loss‚9.91148953831267 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07185770571231842 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5761352181434631 ,jet/accuracy_4_of_4‚0.012545786798000336 .particle/accuracy_3_of_3‚0.999514102935791 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚1143.160185272 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16978827118873596 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5723279714584351 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09479166567325592 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886391162872314 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5337367057800293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5931751728057861 *jet/accuracy_1_of_2‚0.6740078926086426 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5741066336631775 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936507344245911 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9914529323577881 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.43772584199905396 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_2‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6763157248497009 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13119612634181976 +jet/accuracy_2_of_2‚0.11628363281488419 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5732566714286804 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5195426344871521 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8299999833106995 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950514435768127 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6562386751174927 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.40245479345321655 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927893280982971 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07525590062141418 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006514979992061853 particle/accuracy_2_of_2‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7204449772834778 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0642085149884224 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.051620036363601685 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08426923304796219 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14783577620983124 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9180018305778503 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9884023070335388 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978419542312622 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 trainer/global_step‚2402 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21328812837600708 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994543731212616 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5340465307235718 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.847612738609314 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4050000011920929 % _timestamp‚1.7843692017157178e+09 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901549816131592 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5351678133010864 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6527708768844604 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019566211849451065 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.9027735590934753 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06910452246665955 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40123456716537476 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.532484769821167 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5258440971374512 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7465811371803284 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.626311182975769 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7554638981819153 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6196580529212952 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940857887268066 +jet/accuracy_2_of_3‚0.41075998544692993 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5363295674324036 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5074779987335205 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05337774008512497 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035234359093010426 'particle/accuracy‚0.834902822971344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943981170654297 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08188078552484512 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10058094561100006 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.5945784449577332 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06538239866495132 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9944444298744202 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6419483423233032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07422163337469101 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.118327297270298 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11613143980503082 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.25798818469047546 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5959476828575134 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11207333952188492 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8965535759925842 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9768046140670776 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7523182034492493 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125Âì vO‘  ‚(:† ½ íÒ¥ºî‹ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k Ó¡ ‚(:È ½ íÒ×êŒ disk.dm-0.in‚ 1433.4765625 cpu‚26.69607842967748$ memory_percent‚98.71029853820801' proc.memory.availableMB‚ 211.3046875 disk.rbd1.in‚0$ disk.nvme0n1.out‚3641.3701171875 disk.sda1.in‚0 disk.sda2.in‚0! disk.dm-0.out‚3641.3701171875* proc.memory.percent‚16.345882415771484$ disk./.usageGB‚590.2182884216309 disk.rbd1.out‚2.8125 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.10546875 disk.drbd2165.out‚0 network.recv‚402255) disk./.usagePercent‚39.61004456210505 disk.rbd0.out‚0 disk.sda3.in‚ 0.10546875 disk.sda3.out‚ 28.1171875 disk.sda.out‚ 28.1171875! proc.memory.rssMB‚ 2678.109375 disk.nvme0n1.in‚ 1433.4765625 disk.sda2.out‚0 disk.dm-1.out‚ 28.1171875 proc.cpu.threads‚17 network.sent‚ 3.055953e+06" disk.nvme0c0n1.in‚ 1433.4765625& disk.nvme0c0n1.out‚3641.3701171875 disk.dm-1.in‚ 0.10546875 disk.sda1.out‚0Ç9î2¢ - _wandbruntime‚1165  _runtime‚1165~ÊÇÇ£ Á & lr-AdamW/pg1‚0.0008443769346207678 & lr-AdamW/pg2‚0.0008443769346207678 trainer/global_step‚2449 % _timestamp‚1.7843692243217092e+09 _runtime‚1165.765594853 _step‚195dú3Ü™¤ Æ & lr-AdamW/pg1‚0.0008443769346207678 & lr-AdamW/pg2‚0.0008443769346207678 trainer/global_step‚2449 % _timestamp‚1.7843692243217092e+M³S309 _runtime‚1165.765594853 _step‚195Ã<ø{ô2¥ - _wandbruntime‚1165  _runtime‚1165ª<Á¾'¦ ¸O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41665494 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18398981 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚1165.766929572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88292283342848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1441248909310 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1446248052224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚211 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11300988 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 trainer/global_step‚2449 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95259755251200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1495028865024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1446335989248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5582693 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89787480123392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41655515 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95259860227584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93813524238336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843692243229358e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23266511 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95259860227584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936ÚmÎSÃ'§ ½O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93813524238336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843692243229358e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23266511 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95259860227584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41665494 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18398981 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚1165.766929572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88292283342848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1441248909310 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1446248052224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚211 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11300988 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 trainer/global_step‚2449 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95259755251200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1495028865024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1446335989248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5582693 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89787480123392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41655515 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95259860227584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÄΗ¼2¨ - _wandbruntime‚1166  _runtime‚1166eŒâQÇ&© ÁM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18406204 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23265815 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1441730093750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11305469 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18406202 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92273739018240 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95297540558336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88327169953792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95297435574784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16890568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41672017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88327194857472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16890345 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89822784966656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1495789212672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23275806 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11305472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93850618163200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41682010 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5585096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1446905355776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5584876 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1446817411584 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5584876 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1446817411584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95297540558336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88327169953792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95297435574784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16890568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41672017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88327194857472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16890345 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89822784966656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1495789212672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23275806 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11305472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93850618163200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41682010 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5585096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1446905355776źgö2« - _wandbruntime‚1166  _runtime‚1166[b9¬ ‚ 2loss/SRqqt1/assignment_loss‚0.6997757988917064 2loss/FBt2/assignment_loss‚0.013539907842945677 .loss/FRt1/detection_loss‚0.604896165818191 trainer/global_step‚2449 2loss/SRqqt2/assignment_loss‚0.6390749928847629 /loss/FRt2/detection_loss‚1.3045210889464274 &loss/total_loss‚10.021732319513003 _runtime‚1166.116373891 0loss/FRt1/assignment_loss‚2.1112146661695785 0loss/FRt2/assignment_loss‚1.2185731620573608 2loss/FBt1/assignment_loss‚0.013671894314360212 1loss/SRqqt2/detection_loss‚0.6339686155418454 0loss/FBt2/detection_loss‚0.47898415503916386 0loss/SRqqt1/detection_loss‚1.051017540896052 /loss/FBt1/detection_loss‚1.2524943311106076 epoch‚3 $ _timestamp‚1.784369224672838e+09 _step‚198‰Ü½¸ ­ ‡ &loss/total_loss‚10.021732319513003 _runtime‚1166.116373891 0loss/FRt1/assignment_loss‚2.1112146661695785 0loss/FRt2/assignment_loss‚1.2185731620573608 2loss/FBt1/assignment_loss‚0.013671894314360212 1loss/SRqqt2/detection_loss‚0.6339686155418454 0loss/FBt2/detection_loss‚0.47898415503916386 0loss/SRqqt1/detection_loss‚1.051017540896052 /loss/FBt1/detection_loss‚1.2524943311106076 epoch‚3 $ _timestamp‚1.784369224672838e+09 _step‚198 2loss/SRqqt1/assignment_loss‚0.6997757988917064 2loss/FBt2/assignment_loss‚0.013539907842945677 .loss/FRt1/detection_loss‚0.604896165818191 trainer/global_step‚2449 2loss/SRqqt2/assignment_loss‚0.6390749928847629 /loss/FRt2/detection_loss‚1.3045210889464274Æ÷Mœ® ‚(:‘ Ì íÒ¡øä‹ gpu.0.gpu‚32.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ÷zÜЯ ‚(:Å Ì íÒ©Ñ’ disk.rbd1.in‚0 disk.nvme0n1.in‚ 1433.4765625 disk.rbd0.out‚0$ memory_percent‚98.70953559875488! proc.memory.rssMB‚ 2678.109375 cpu‚26.705088454618867$ disk./.usageGB‚590.2179069519043' proc.memory.availableMB‚ 211.4296875 disk.sda1.out‚0 disk.sda.in‚ 0.10546875 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚3710.5791015625 disk.dm-1.in‚ 0.10546875! disk.dm-0.out‚3710.5791015625 disk.drbd2165.out‚0 disk.rbd1.out‚2.875" disk.nvme0c0n1.in‚ 1433.4765625 disk.dm-0.in‚ 1433.4765625 disk.sda3.in‚ 0.10546875$ disk.nvme0n1.out‚3710.5791015625 disk.sda3.out‚ 28.453125 network.sent‚ 3.113181e+06 disk.dm-1.out‚ 28.453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 28.453125* proc.memory.percent‚16.345882415771484 network.recv‚407345) disk./.usagePercent‚39.61001896135163 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17ƒÃ•rœ° ‚(:‘ Û íÒý©ÞŠ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚171.223' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.244600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ýѱ ‚(:Æ Û íÒàäóŒ disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚ 28.76171875 disk.dm-0.out‚3713.681640625 disk.sda3.in‚ 0.10546875 disk.sda.in‚ 0.10546875 disk.drbd2165.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.345882415771484 disk.sda.out‚ 28.76171875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 1491.859375# disk.nvme0n1.out‚3713.681640625 disk.sda1.in‚0! proc.memory.rssMB‚ 2678.109375 proc.cpu.threads‚17 network.sent‚ 3.175825e+06 network.recv‚412983 disk.rbd1.out‚2.875! disk.nvme0c0n1.in‚ 1491.859375 disk.dm-0.in‚ 1491.859375 cpu‚26.712252071336724) disk./.usagePercent‚39.61001460922355$ disk./.usageGB‚590.2178421020508% disk.nvme0c0n1.out‚3713.681640625$ memory_percent‚98.71127605438232( proc.memory.availableMB‚ 211.14453125 disk.sda3.out‚ 28.76171875 disk.rbd0.in‚0 disk.rbd0.out‚0”¾ó2² - _wandbruntime‚1188  _runtime‚1188Ä]Þdz Á _runtime‚1188.911678654 _step‚199 & lr-AdamW/pg1‚0.0008441503625337852 & lr-AdamW/pg2‚0.0008441503625337852 trainer/global_step‚2499 % _timestamp‚1.7843692474677894e+09µ:sÌ´ Æ & lr-AdamW/pg1‚0.0008441503625337852 & lr-AdamW/pg2‚0.0008441503625337852 trainer/global_step‚2499 % _timestamp‚1.7843692474677894e+09 _runtime‚1188.911678654 _step‚199Çohé2µ - _wandbruntime‚1188  _runtime‚1188Áðøš½'¶ ·O IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90037834170368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97143908663808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95669055410176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1474836214272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91562263368192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚95530515638330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18760033 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1188.913208331 _step‚200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95669072449536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18760031 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17217712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5692760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90037859074048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18760033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1524613890560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97143803687424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11525188 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97143803687424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23732413 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 trainer/global_step‚2499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1474748277248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 $ _timestamp‚1.784369247469085e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95530412461880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1469560751928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23722436 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97143908663808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17217948 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94060868749312 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23732413 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11525185 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91562472964608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473H*ÔÌî&· ¼O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42482467 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5692527 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90037834170368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97143908663808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95669055410176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1474836214272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91562263368192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚95530515638330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18760033 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1188.913208331 _step‚200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95669072449536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18760031 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17217712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5692760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90037859074048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18760033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1524613890560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97143803687424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11525188 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97143803687424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23732413 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 trainer/global_step‚2499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1474748277248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 $ _timestamp‚1.784369247469085e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95530412461880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1469560751928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23722436 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97143908663808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17217948 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94060868749312 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23732413 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11525185 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91562472964608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1475405580800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5694936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97181588994560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91597964289536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470214210660 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23741708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11529672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94097340784640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5694701 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90072770588672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95567451818738 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78436924783097e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42508962 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95706183413760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1470128073458 _runtime‚1189.275097407 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1525019501056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94097323745280 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42508962 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91597765186048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1475405580800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23731717 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42498969 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18767252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95706166374400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17224370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97181588994560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11529669 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42498969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95706166374400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18767254 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23731717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192Hï‰Ë&º ÅM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23731717 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42498969 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18767252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95706166374400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17224370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97181588994560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11529669 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42498969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95706166374400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18767254 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23731717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1475405580800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193107968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5694936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97181588994560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91597964289536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470214210660 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23741708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11529672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94097340784640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5694701 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90072770588672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95567451818738 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78436924783097e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42508962 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95706183413760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1470128073458 _runtime‚1189.275097407 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1525019501056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94097323745280 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42508962 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91597765186048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1475405580800ÉI÷ÝÏ2» - _wandbruntime‚1189  _runtime‚1189•Ì@¼ ÿ /loss/FBt1/detection_loss‚0.6895032881034745 epoch‚3 _step‚202 2loss/SRqqt1/assignment_loss‚0.7392793965007135 /loss/FRt2/detection_loss‚1.2409418614941738 /loss/FBt2/detection_loss‚0.5641662934335487 %loss/total_loss‚9.566549969663043 trainer/global_step‚2499 $ _timestamp‚1.784369247832412e+09 _runtime‚ 1189.27607902 0loss/FRt2/assignment_loss‚1.1648716386073807 2loss/SRqqt2/assignment_loss‚0.5862568081081758 1loss/FBt1/assignment_loss‚0.20323237870535296 1loss/SRqqt1/detection_loss‚1.0880343026462818 1loss/SRqqt2/detection_loss‚0.6257853746656279 /loss/FRt1/assignment_loss‚2.021523112204017 2loss/FBt2/assignment_loss‚0.014204032012383482 /loss/FRt1/detection_loss‚0.6287514831819137¥¶'´ ½ „ %loss/total_loss‚9.566549969663043 trainer/global_step‚2499 $ _timestamp‚1.784369247832412e+09 _runtime‚ 1189.27607902 0loss/FRt2/assignment_loss‚1.1648716386073807 2loss/SRqqt2/assignment_loss‚0.5862568081081758 1loss/FBt1/assignment_loss‚0.20323237870535296 1loss/SRqqt1/detection_loss‚1.0880343026462818 1loss/SRqqt2/detection_loss‚0.6257853746656279 /loss/FRt1/assignment_loss‚2.021523112204017 2loss/FBt2/assignment_loss‚0.014204032012383482 /loss/FRt1/detection_loss‚0.6287514831819137 /loss/FBt1/detection_loss‚0.6895032881034745 epoch‚3 _step‚202 2loss/SRqqt1/assignment_loss‚0.7392793965007135 /loss/FRt2/detection_loss‚1.2409418614941738 /loss/FBt2/detection_loss‚0.5641662934335487Ê,9©‘¾ ‚(:† ê íÒ¿´ýŠ gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚178.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~éÝ¿ ‚(:· ê íÒªÛêŒ disk.sda1.out‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2678.109375* proc.memory.percent‚16.345882415771484 disk.sda3.out‚ 28.8515625! disk.nvme0c0n1.out‚ 3719.03125' proc.memory.availableMB‚ 211.3515625 disk.dm-1.in‚ 0.10546875 disk.sda1.in‚0 disk.dm-0.in‚ 1491.859375 disk.drbd2165.in‚0 disk.rbd1.in‚0 network.recv‚416083$ disk./.usageGB‚590.2179908752441 disk.nvme0n1.in‚ 1491.859375 disk.dm-0.out‚ 3719.03125 disk.sda3.in‚ 0.10546875 disk.sda.out‚ 28.8515625 cpu‚26.721676609033075 disk.nvme0n1.out‚ 3719.03125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.71001243591309 disk.dm-1.out‚ 28.8515625 disk.rbd1.out‚2.9375 disk.sda2.out‚0 network.sent‚ 3.180829e+06) disk./.usagePercent‚39.61002459351738 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.10546875! disk.nvme0c0n1.in‚ 1491.859375ꈪš2À - _wandbruntime‚1211  _runtime‚1211Ñ©÷ˆÇÁ Á trainer/global_step‚2549 % _timestamp‚1.7843692703716714e+09 _runtime‚1211.815669834 _step‚203 & lr-AdamW/pg1‚0.0008439170613722093 & lr-AdamW/pg2‚0.0008439170613722093(_qÑÌ Æ & lr-AdamW/pg1‚0.0008439170613722093 & lr-AdamW/pg2‚0.0008439170613722093 trainer/global_step‚2549 % _timestamp‚1.7843692703716714e+09 _runtime‚1211.815669834 _step‚203Ëžÿ3Ð2à - _wandbruntime‚1211  _runtime‚1211ÕËÓ|#Ä ¸O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1554026808832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97524603621376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5801771 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91783434805248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1498044868726 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _step‚204 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1503248502272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97524603621376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97382515384438 % _timestamp‚1.7843692703731077e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24188338 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1503336439296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5801996 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99027957100032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1497958731636 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95884453476352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93337258309120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99027852123648 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99027852123648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95884470515712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43309419 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1503248502272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17551384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24198315 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17551156 HBDe},^BviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93337258309120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99027852123648 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99027852123648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95884470515712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43309419 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1503248502272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17551384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24198315 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17551156 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97524603621376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97382515384438 % _timestamp‚1.7843692703731077e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24188338 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1503336439296 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5801996 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977Ìrz892Æ - _wandbruntime‚1212  _runtime‚1212oK*È&Ç ÂM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97561714585600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99065532447232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24207610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1554611692032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97419554741408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43325921 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43335914 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19128302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97561731624960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1554433297920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1498526053166 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91818321416192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17558095 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95920925511680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43335914 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19128304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24197619 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11753872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93372754714112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚205 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19128304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11753869 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97561731624960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97561714585600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17557836 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1503905805824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91818346319872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24207610 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99065637430784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93372958011904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552™º Í&È ÇM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24207610 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99065637430784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93372958011904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97561714585600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99065532447232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24207610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1554611692032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97419554741408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43325921 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43335914 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19128302 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97561731624960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1554433297920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1498526053166 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91818321416192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17558095 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95920925511680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43335914 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19128304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24197619 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11753872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93372754714112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚205 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19128304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11753869 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97561731624960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97561714585600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17557836 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1503905805824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91818346319872Í—Öµ‚2É - _wandbruntime‚1212  _runtime‚1212û¡žÊ € .loss/FBt2/detection_loss‚0.516453188020558 2loss/SRqqt1/assignment_loss‚0.9394676733102082 trainer/global_step‚2549 _runtime‚1212.179572529 epoch‚3 % _timestamp‚1.7843692707356765e+09 2loss/FBt2/assignment_loss‚0.017177653970962832 1loss/SRqqt1/detection_loss‚1.0622658463725387 /loss/FBt1/detection_loss‚0.5235877506434917 %loss/total_loss‚9.583324500335554 _step‚206 /loss/FRt1/assignment_loss‚2.009706634023915 0loss/FRt2/assignment_loss‚1.1066023170735912 2loss/SRqqt2/assignment_loss‚0.6287491453070996 2loss/FBt1/assignment_loss‚0.005663699608157912 .loss/FRt1/detection_loss‚0.643900148038665 /loss/FRt2/detection_loss‚1.3666678212245005 1loss/SRqqt2/detection_loss‚0.7630826227418664óW+ÄË … /loss/FBt1/detection_loss‚0.5235877506434917 %loss/total_loss‚9.583324500335554 _step‚206 /loss/FRt1/assignment_loss‚2.009706634023915 0loss/FRt2/assignment_loss‚1.1066023170735912 2loss/SRqqt2/assignment_loss‚0.6287491453070996 2loss/FBt1/assignment_loss‚0.005663699608157912 .loss/FRt1/detection_loss‚0.643900148038665 /loss/FRt2/detection_loss‚1.3666678212245005 1loss/SRqqt2/detection_loss‚0.7630826227418664 .loss/FBt2/detection_loss‚0.516453188020558 2loss/SRqqt1/assignment_loss‚0.9394676733102082 trainer/global_step‚2549 _runtime‚1212.179572529 epoch‚3 % _timestamp‚1.7843692707356765e+09 2loss/FBt2/assignment_loss‚0.017ˆ²ÜÕG177653970962832 1loss/SRqqt1/detection_loss‚1.0622658463725387άø+%Ì ‚(:„ ù íÒùöäŠ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚172.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*-$¯ÓÍ ‚(:È ù íÒÌË«$ disk./.usageGB‚590.2179107666016 disk.sda3.out‚ 29.03515625 disk.drbd2165.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2678.109375 disk.sda3.in‚ 0.10546875 disk.drbd2165.out‚0 disk.dm-1.out‚ 29.03515625! disk.dm-0.out‚3722.0263671875 cpu‚26.72843420770632 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 1491.859375 network.sent‚ 3.243534e+06 disk.sda.in‚ 0.10546875 disk.sda2.in‚0* proc.memory.percent‚16.345882415771484 proc.cpu.threads‚17) disk./.usagePercent‚39.61001921735916 disk.sda.out‚ 29.03515625 disk.sda1.in‚0# memory_percent‚98.7114667892456& disk.nvme0c0n1.out‚3722.0263671875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 1491.859375 disk.nvme0n1.in‚ 1491.859375$ disk.nvme0n1.out‚3722.0263671875( proc.memory.availableMB‚ 211.11328125 network.recv‚421395 disk.rbd1.in‚0 disk.rbd1.out‚2.9375 disk.rbd0.in‚0 disk.sda2.out‚04t/Î ‚(:„ ˆ¡íÒáàú‹ gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚184.52' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ ¬fÂÏ ‚(:· ˆ¡íÒœþÊŒ disk.sda3.out‚ 29.421875 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 1491.859375 disk.dm-0.out‚ 3726.5078125 disk.drbd2165.in‚0* proc.memory.percent‚16.345882415771484 cpu‚26.737073003768632 network.sent‚ 3.306059e+06# disk.nvme0c0n1.out‚ 3726.5078125 disk.rbd0.in‚0# memory_percent‚98.7100601196289 disk.rbd1.out‚3.0625 proc.cpu.threads‚17% proc.memory.availableMB‚ 211.34375$ disk./.usageGB‚590.2173728942871 disk.sda3.in‚ 0.10546875 disk.sda.out‚ 29.421875! disk.nvme0n1.out‚ 3726.5078125 disk.rbd0.out‚0 disk.dm-0.in‚ 1491.859375 disk.sda.in‚ 0.10546875 disk.nvme0n1.in‚ 1491.859375 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 29.421875! proc.memory.rssMB‚ 2678.109375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875 network.recv‚426291) disk./.usagePercent‚39.60998312029683 disk.rbd1.in‚0(g2Ð - _wandbruntime‚1235  _runtime‚1235òõ³ÇÑ Á & lr-AdamW/pg1‚0.0008436770348728318 & lr-AdamW/pg2‚0.0008436770348728318 trainer/global_step‚2599 % _timestamp‚1.7843692936257463e+09 _runtime‚1235.069605626 _step‚207©’ìòÌÒ Æ & lr-AdamW/pg2‚0.0008436770348728318 trainer/global_step‚2599 % _timestamp‚1.7843692936257463e+09 _runtime‚1235.069605626 _step‚207 & lr-AdamW/pg1‚0.0008436770348728318Ï\oþ×2Ó - _wandbruntime‚1235  _runtime‚1235>¥ð7Â'Ô ¼O _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97708072282112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1583504058368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19482131 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99380168871936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24664217 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95112315484160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99234411954096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19482133 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17884614 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93528985632768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100912005536256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100911900559872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1531748727296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93529010536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97708055242752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1531836664320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44146350 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24654240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1531748727296 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17884395 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11973588 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1583329851392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100911900559872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19482131 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1526442848434 % _timestamp‚1.7843692936268826e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44136371 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11973585 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24654240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5910810 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99380168871936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99380151832576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24664217 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _runtime‚1235.070859873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1531836664320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95112514594816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1526356711344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5911026 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99380151832576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100911900559872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19482131 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1526442848434 % _timestamp‚1.7843692936268826e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44136371 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11973585 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24654240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5910810 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99380168871936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99380151832576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24664217 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _runtime‚1235.070859873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1531836664320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95112514594816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1526356711344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5911026 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99380151832576 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99234411954096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19482133 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17884614 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93528985632768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100912005536256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100911900559872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1531748727296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93529010536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97708055242752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1531836664320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44146350 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24654240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1531748727296 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17884395 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11973588 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÐÀöU2Ö - _wandbruntime‚1235  _runtime‚1235ŒÏèeù"× ÅM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93563922051072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚209 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1584091296768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97744527278080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97744544317440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44162866 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99417262796800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24663521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11978072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95148013347840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100949685867008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19489352 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95147820535808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24673512 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100949685867008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44152873 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19489352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99417262796800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1235.418088652 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1532406030848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24673512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17890991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93563897147392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19489354 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1532406030848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44152873 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5912922 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99271554487516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1583923388416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1532318086656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99271451310954 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473Áìà‘Ð&Ø ÊM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99417262796800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24663521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11978072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95148013347840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100949685867008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19489352 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95147820535808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24673512 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100949685867008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44152873 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19489352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99417262796800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1235.418088652 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1532406030848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24673512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17890991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93563897147392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19489354 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1532406030848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44152873 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5912922 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99271554487516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1583923388416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1532318086656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99271451310954 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93563922051072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚209 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1584091296768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97744527278080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97744544317440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44162866 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192Ñ%Zس2Ù - _wandbruntime‚1235  _runtime‚12355iy[Ú ‚ 0loss/SRqqt1/detection_loss‚1.165297280310105 1loss/SRqqt2/detection_loss‚0.7806657183303289 /loss/FBt1/detection_loss‚0.6215473839081824 /loss/FBt2/detection_loss‚0.4395712273995536 epoch‚3 % _timestamp‚1.7843692939759798e+09 0loss/FRt1/assignment_loss‚1.9926411254065377 2loss/SRqqt1/assignment_loss‚0.6851874500024298 /loss/FRt2/detection_loss‚1.4036082698519088 %loss/total_loss‚9.554614593854268 trainer/global_step‚2599 _runtime‚1235.419523581 _step‚210 2loss/FBt2/assignment_loss‚0.009597068933835629 /loss/FRt1/detection_loss‚0.6122097087738314 2loss/SRqqt2/assignment_loss‚0.5895202739499628 2loss/FBt1/assignment_loss‚0.005977508394914821 0loss/FRt2/assignment_loss‚1.2487915785926762BIÝ Û ‡ 2loss/SRqqt2/assignment_loss‚0.5895202739499628 2loss/FBt1/assignment_loss‚0.005977508394914821 0loss/FRt2/assignment_loss‚1.2487915785926762 0loss/SRqqt1/detection_loss‚1.165297280310105 1loss/SRqqt2/detection_loss‚0.7806657183303289 /loss/FBt1/detection_loss‚0.6215473839081824 /loss/FBt2/detection_loss‚0.4395712273995536 epoch‚3 % _timestamp‚1.7843692939759798e+09 0loss/FRt1/assignment_loss‚1.9926411254065377 2loss/SRqqt1/assignment_loss‚0.6851874500024298 /loss/FRt2/detection_loss‚1.4036082698519088 %loss/total_loss‚9.554614593854268 trainer/global_step‚2599 _runtime‚1235.419523581 _step‚210 2loss/FBt2/assignment_loss‚0.009597068933835629 /loss/FRt1/detection_loss‚0.6122097087738314Ò4wô¨Ü ‚(:… —¡íÒæà˜‹ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚179.098' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ èžØÝ ‚(:Í —¡íÒ¾Ž‹ disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1571.59765625 network.sent‚ 3.311154e+06) disk./.usagePercent‚39.61010958801875$ disk.nvme0n1.out‚3729.6220703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 29.8046875 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.out‚3.0625$ disk./.usageGB‚590.2192573547363 disk.dm-0.in‚ 1571.59765625 disk.sda.in‚ 0.10546875 disk.sda.out‚ 29.8046875 cpu‚26.744078509630125 network.recv‚429890! disk.nvme0n1.in‚ 1571.59765625 disk.sda3.in‚ 0.10546875! disk.dm-0.out‚3729.6220703125 disk.sda3.out‚ 29.8046875 disk.drbd2165.out‚0$ memory_percent‚98.71151447296143* proc.memory.percent‚16.345882415771484 disk.sda1.in‚0( proc.memory.availableMB‚ 211.10546875! proc.memory.rssMB‚ 2678.109375& disk.nvme0c0n1.out‚3729.6220703125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875ŽC5Á2Þ - _wandbruntime‚1258  _runtime‚1258LTsÇß Á % _timestamp‚1.7843693165753672e+09 _runtime‚1258.019254722 _step‚211 & lr-AdamW/pg1‚0.0008434302868801642 & lr-AdamW/pg2‚0.0008434302868801642 trainer/global_step‚2649’\Ìà Æ & lr-AdamW/pg1‚0.0008434302868801642 & lr-AdamW/pg2‚0.0008434302868801642 trainer/global_step‚2649 % _timestamp‚1.7843693165753672e+09 _runtime‚1258.019254722 _step‚211Ó­ 2á - _wandbruntime‚1258  _runtime‚1258À¥ÁÈ'â ÂO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101235700043776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1613002562560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1560248952320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25120142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18216542 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102795948996096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6018989 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44963323 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6018757 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚101235700043776 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12197785 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚101086514876654 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1554840828142 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44973302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19843183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101235717083136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚101086411700204 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101235717083136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚212 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19843181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18216777 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25130119 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102796053972480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25130119 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96887588830208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19843181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25120142¯pVò•ã ÇO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96887588830208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19843181 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25120142 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101235700043776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1613002562560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1560248952320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25120142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18216542 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102795948996096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6018989 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44963323 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6018757 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚101235700043776 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12197785 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚101086514876654 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1554840828142 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44973302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19843183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101235717083136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 UADeviceS›8tatsMonitor.on_train_batch_start/requested_bytes.all.freed‚101086411700204 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101235717083136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚212 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19843181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18216777 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25130119 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102796053972480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25130119 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473Ô1¢×2ä - _wandbruntime‚1258  _runtime‚12580àÖ½Ñ&å ËM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101272828047360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19850404 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1560818311680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101272828047360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44979825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12202269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19850404 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚101123554233624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1555408149784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6021104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚1258.385104993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102833734303232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18223141 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95309472878592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44989818 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19850402 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25129423 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102833629319680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12202272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1613585754112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25129423 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1560818311680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19850402 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96923083536384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99568129044480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102833734303232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101272811008000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44989818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1613413651456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1555322012582 $ _timestamp‚1.784369316940841e+09 _step‚213 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44979825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101272811008000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6020872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18223376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96922886530048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚101123451057062 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146Æ”æÆÖ&æ ÐM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95309472878592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44989818 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19850402 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25129423 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102833629319680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12202272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1613585754112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25129423 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1560818311680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19850402 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96923083536384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99568129044480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102833734303232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101272811008000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44989818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1613413651456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1555322012582 $ _timestamp‚1.784369316940841e+09 _step‚213 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44979825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101272811008000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6020872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18223376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96922886530048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚101123451057062 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101272828047360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19850404 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1560818311680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101272828047360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44979825 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12202269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19850404 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚101123554233624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1555408149784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6021104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚1258.385104993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102833734303232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18223141 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473ÕEc;2ç - _wandbruntime‚1258  _runtime‚1258à© è … 0loss/FRt1/assignment_loss‚2.0058107629970268 2loss/SRqqt1/assignment_loss‚0.7675968409804949 3loss/FBt1/assignment_loss‚0.0035544746308150106 2loss/FBt2/assignment_loss‚0.014292659340325232 % _timestamp‚1.7843693169424093e+09 /loss/FRt1/detection_loss‚0.5893406107269707 1loss/SRqqt1/detection_loss‚1.0176201085206764 1loss/SRqqt2/detection_loss‚0.8737484072453704 _runtime‚1258.386060265 2loss/SRqqt2/assignment_loss‚0.6832010762715581 /loss/FBt1/detection_loss‚0.7851666709916159 epoch‚3 _step‚214 0loss/FRt2/assignment_loss‚1.3366915758233517 /loss/FRt2/detection_loss‚1.1876858497716967 /loss/FBt2/detection_loss‚0.7768492596510511 &loss/total_loss‚10.041558296950953 trainer/global_step‚2649 aÁãé Š trainer/global_step‚2649 0loss/FRt1/assignment_loss‚2.0058107629970268 2loss/SRqqt1/assignment_loss‚0.7675968409804949 3loss/FBt1/assignment_loss‚0.0035544746308150106 2loss/FBt2/assignment_loss‚0.014292659340325232 % _timestamp‚1.7843693169424093e+09 /loss/FRt1/detection_loss‚0.5893406107269707 1loss/SRqqt1/detection_loss‚1.0176201085206764 1loss/SRqqt2/detection_loss‚0.8737484072453704 _runtime‚1258.386060265 2loss/SRqqt2/assignment_loss‚0.6832010762715581 /loss/FBt1/detection_loss‚0.7851666709916159 epoch‚3 _step‚214 0loss/FRt2/assignment_loss‚1.3366915758233517 /loss/FRt2/detection_loss‚1.1876858497716967 /loss/FBt2/detection_loss‚0.7768492596510511 &loss/total_loss‚10.041558296950953ÖOj¤=‘ê ‚(:† ¦¡íÒ¡˜¹Š gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ð†Ôë ‚(:É ¦¡íÒÒƒÛŒ# memory_percent‚98.7119197845459 network.recv‚434823 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 3733.89453125 disk.sda3.out‚ 30.05859375 disk.sda.in‚ 0.10546875 proc.cpu.threads‚17 network.sent‚ 3.373827e+06# disk.nvme0c0n1.in‚ 1571.59765625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2678.11328125) disk./.usagePercent‚39.61010165178519 disk.rbd0.out‚0 disk.dm-0.in‚ 1571.59765625* proc.memory.percent‚16.345906257629395 disk.sda.out‚ 30.05859375 disk.sda1.out‚0 disk.dm-1.out‚ 30.05859375$ disk./.usageGB‚590.2191390991211 disk.rbd1.out‚3.125" disk.nvme0n1.out‚ 3733.89453125 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0' proc.memory.availableMB‚ 211.0390625 cpu‚26.752123943016965 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 3733.89453125 disk.dm-1.in‚ 0.10546875! disk.nvme0n1.in‚ 1571.59765625f:ñ_ì ‚(:… µ¡íÒå·´Š gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©«Ðí ‚(:Å µ¡íҘɢ disk.sda.in‚ 0.10546875 disk.drbd2165.in‚0 disk.dm-1.out‚30.3125) disk./.usagePercent‚39.61013723683245 disk.rbd0.out‚0( proc.memory.availableMB‚ 211.34765625 cpu‚26.758601250345237 disk.rbd1.in‚0 disk.sda.out‚30.3125# proc.memory.rssMB‚ 2678.11328125 network.recv‚440218 disk.drbd2165.out‚0 disk.rbd1.out‚3.125% disk.nvme0n1.out‚3736.98486328125# disk./.usageGB‚590.219669342041" disk.dm-0.out‚3736.98486328125' disk.nvme0c0n1.out‚3736.98486328125 disk.sda2.in‚0" memory_percent‚98.710036277771* proc.memory.percent‚16.345906257629395 proc.cpu.threads‚17 disk.dm-0.in‚ 1571.59765625 disk.sda3.in‚ 0.10546875! disk.nvme0n1.in‚ 1571.59765625 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 3.436331e+06# disk.nvme0c0n1.in‚ 1571.59765625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.out‚30.3125„›/2î - _wandbruntime‚1281  _runtime‚1281JÏ23Çï Á % _timestamp‚1.7843693398497097e+09 _runtime‚1281.293606818 _step‚215 & lr-AdamW/pg1‚0.0008431768213463765 & lr-AdamW/pg2‚0.0008431768213463765 trainer/global_step‚2699’ÑFaÌð Æ _step‚215 & lr-AdamW/pg1‚0.0008431768213463765 & lr-AdamW/pg2‚0.0008431768213463765 trainer/global_step‚2699 % _timestamp‚1.7843693398497097e+09 _runtime‚1281.293606818מô© 2ñ - _wandbruntime‚1281  _runtime‚1281€èUvò ÃO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104680102408704 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103091248254976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18547864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101355258775552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20204231 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104679997432320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1642304348672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12421988 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98662642651648 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18548106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1583152670760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97020137095168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102938514622762 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97020161998848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101355275814912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104679997432320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 $ _timestamp‚1.784369339850866e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1588749177344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98662441443840 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18548106 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1583152670760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97020137095168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102938514622762 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97020161998848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101355275814912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104679997432320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 $ _timestamp‚1.784369339850866e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1588749177344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98662441443840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104680102408704 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103091248254976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18547864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101355258775552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20204231 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104679997432320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1642304348672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ØmÓ2ô - _wandbruntime‚1281  _runtime‚1281†!ÖÔ&õ ÎM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45816770 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102975553979732 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104717782739456 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12426472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1643068943872 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18554510 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 _step‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25595325 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12426469 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20211454 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6128276 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45806777 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1583806129492 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25595325 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97055048609792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103128359219200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45816770 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101391747850240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20211454 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103128376258560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25605316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6128041 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103128376258560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102975450803170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98698142457344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101391730810880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98697939159552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 trainer/global_step‚2699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25605316 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1642890549760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18554510 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 _step‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25595325 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12426469 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20211454 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6128276 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45806777 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1583806129492 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25595325 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97055048609792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103128359219200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45816770 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101391747850240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20211454 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103128376258560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25605316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6128041 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103128376258560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102975450803170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98698142457344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101391730810880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98697939159552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 trainer/global_step‚2699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25605316 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1642890549760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12426472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1643068943872 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192Ùv›™2÷ - _wandbruntime‚1281  _runtime‚128143 øø ý 0loss/FRt1/assignment_loss‚2.0952644089812886 0loss/FBt1/assignment_loss‚0.2674905579905338 .loss/FBt2/detection_loss‚0.513797880852149 &loss/total_loss‚10.131394452555142 trainer/global_step‚2699 _step‚218 2loss/SRqqt1/assignment_loss‚0.7540856329168981 0loss/FBt2/assignment_loss‚0.0382843622924126 /loss/FRt1/detection_loss‚0.5625878013619201 .loss/FRt2/detection_loss‚1.402925008815536 1loss/SRqqt1/detection_loss‚1.0594130142233693 epoch‚3 /loss/FRt2/assignment_loss‚1.411545839386913 2loss/SRqqt2/assignment_loss‚0.6151115631315438 1loss/SRqqt2/detection_loss‚0.8142650745224935 /loss/FBt1/detection_loss‚0.5966233080800842 % _timestamp‚1.7843693401941783e+09 _runtime‚1281.638079812âwwù ‚ _step‚218 2loss/SRqqt1/assignment_loss‚0.7540856329168981 0loss/FBt2/assignment_loss‚0.0382843622924126 /loss/FRt1/detection_loss‚0.5625878013619201 .loss/FRt2/detection_loss‚1.402925008815536 1loss/SRqqt1/detection_loss‚1.0594130142233693 epoch‚3 /loss/FRt2/assignment_loss‚1.411545839386913 2loss/SRqqt2/assignment_loss‚0.6151115631315438 1loss/SRqqt2/detection_loss‚0.8142650745224935 /loss/FBt1/detection_loss‚0.5966233080800842 % _timestamp‚1.7843693401941783e+09 _runtime‚1281.638079812 0loss/FRt1/assignment_loss‚2.0952644089812886 0loss/FBt1/assignment_loss‚0.2674905579905338 .loss/FBt2/detection_loss‚0.513797880852149 &loss/total_loss‚10.131394452555142 trainer/global_step‚2699Úö"÷,›ú ‚(: Ä¡íÒÉ­ä‹ gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚178.569' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.71379999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08TÄÂÝû ‚(:Ò Ä¡íÒ„ýª) disk./.usagePercent‚39.61038761220093 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 30.5859375% disk.nvme0n1.out‚3743.03759765625 disk.dm-1.in‚ 0.10546875" disk.dm-0.out‚3743.03759765625$ disk./.usageGB‚590.2234001159668! disk.nvme0n1.in‚ 1571.59765625 disk.sda1.in‚0 disk.sda.in‚ 0.10546875$ memory_percent‚98.71160984039307# proc.memory.rssMB‚ 2678.11328125* proc.memory.percent‚16.345906257629395 disk.dm-0.in‚ 1571.59765625 disk.sda3.in‚ 0.10546875 network.recv‚443706 disk.rbd1.out‚3.1875 disk.rbd0.out‚0 network.sent‚ 3.441321e+06# disk.nvme0c0n1.in‚ 1571.59765625 disk.sda3.out‚ 30.5859375 disk.sda.out‚ 30.5859375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 211.08984375' disk.nvme0c0n1.out‚3743.03759765625 disk.sda2.in‚0 cpu‚26.764951437890357©ÄÒ2ü - _wandbruntime‚1304  _runtime‚1304?ãoÅý ¿ _step‚219 & lr-AdamW/pg1‚0.0008429166423312338 & lr-AdamW/pg2‚0.0008429166423312338 trainer/global_step‚2749 # _timestamp‚1.78436936291182e+09 _runtime‚1304.355645986lÊþ Ä & lr-AdamW/pg1‚0.0008429166423312338 & lr-AdamW/pg2‚0.0008429166423312338 trainer/global_step‚2749 # _timestamp‚1.78436936291182e+09 _runtime‚1304.355645986 _step‚219ÛqÞ]˜2ÿ - _wandbruntime‚1304  _runtime‚1304"”Ì'€ ÆO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106564045868544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20565281 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6235049 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12646185 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104946813505536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26051946 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46627206 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98765737730048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚220 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98765712826368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100437517591552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26061923 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26061923 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1671804765184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104946796466176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20565283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12646188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103178860541952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20565283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1611550650468 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106564045868544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1617249402368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18881457 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6235269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100437708313600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20565281 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106564150844928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26051946 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104946796466176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103178877581312 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46617227 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104946813505536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143i[sSÑ' ËO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106564045868544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1617249402368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18881457 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6235269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100437708313600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20565281 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106564150844928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26051946 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104946796466176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103178877581312 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46617227 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104946813505536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106564045868544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20565281 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6235049 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12646185 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104946813505536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26051946 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46627206 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98765737730048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚220 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98765712826368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100437517591552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26061923 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26061923 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1671804765184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104946796466176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20565283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12646188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103178860541952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20565283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1611550650468 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346ÜÞß}¦2‚ - _wandbruntime‚1304  _runtime‚1304v.#„ƒ ÐM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98800624340992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26071218 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚221 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26061227 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106601726192128 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1617818761728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20572502 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18888112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚1304.701774824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106601831175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98800649244672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1612204109200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18887880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46643722 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46633729 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 trainer/global_step‚2749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12650669 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104983924469760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1617906705920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1617818761728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1612117971998 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103215349616640 ¾nÛ&„ ÕM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1612204109200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18887880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46643722 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46633729 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 trainer/global_step‚2749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12650669 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104983924469760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1617906705920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1617818761728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1612117971998 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103215349616640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98800624340992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26071218 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚221 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26061227 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106601726192128 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1617818761728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20572502 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18888112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚1304.701774824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106601831175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98800649244672Ý%¼2… - _wandbruntime‚1304  _runtime‚1304ãÀv: † „ /loss/FRt1/detection_loss‚0.6264616968205869 1loss/SRqqt1/detection_loss‚1.0413812233747302 0loss/FBt1/detection_loss‚0.49428297380606334 /loss/FBt2/detection_loss‚0.8052752914086899 epoch‚3 0loss/FRt2/assignment_loss‚1.2049178478827003 0loss/SRqqt1/assignment_loss‚0.77380977226812 2loss/SRqqt2/assignment_loss‚0.6093137990664373 3loss/FBt1/assignment_loss‚0.0003543647554730948 3loss/FBt2/assignment_loss‚0.0025096068746024046 % _timestamp‚1.7843693632589202e+09 0loss/FRt1/assignment_loss‚2.1283010378050706 /loss/FRt2/detection_loss‚1.2704862387413656 _step‚222 1loss/SRqqt2/detection_loss‚0.7302024015335685 %loss/total_loss‚9.687296254337408 trainer/global_step‚2749 _runtime‚1304.702800039¢`0ȇ ‰ 1loss/SRqqt2/detection_loss‚0.7302024015335685 %loss/total_loss‚9.687296254337408 trainer/global_step‚2749 _runtime‚1304.702800039 /loss/FRt1/detection_loss‚0.6264616968205869 1loss/SRqqt1/detection_loss‚1.0413812233747302 0loss/FBt1/detection_loss‚0.49428297380606334 /loss/FBt2/detection_loss‚0.8052752914086899 epoch‚3 0loss/FRt2/assignment_loss‚1.2049178478827003 0loss/SRqqt1/assignment_loss‚0.77380977226812 2loss/SRqqt2/assignment_loss‚0.6093137990664373 3loss/FBt1/assignment_loss‚0.0003543647554730948 3loss/FBt2/assignment_loss‚0.0025096068746024046 % _timestamp‚1.7843693632589202e+09 0loss/FRt1/assignment_loss‚2.1283010378050706 /loss/FRt2/detection_loss‚1.2704862387413656 _step‚222Þ¸QØuŽˆ ‚(:ƒ Ó¡íÒŒôà gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚28.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú]_@؉ ‚(:Í Ó¡íÒŽñÄ’ network.recv‚448871" disk.dm-0.out‚3750.55810546875 disk.sda3.in‚ 0.10546875 disk.sda3.out‚ 30.7109375% disk.nvme0n1.out‚3750.55810546875 network.sent‚ 3.516699e+06) disk./.usagePercent‚39.61092676406805$ disk./.usageGB‚590.2314338684082 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 30.7109375 disk.drbd2165.out‚0 disk.rbd1.out‚3.1875" disk.nvme0c0n1.in‚ 1659.6953125 disk.dm-0.in‚ 1659.6953125* proc.memory.percent‚16.345906257629395 disk.sda.out‚ 30.7109375 disk.drbd2165.in‚0( proc.memory.availableMB‚ 211.16796875 cpu‚26.7715049817905 proc.cpu.threads‚17 disk.sda1.in‚0' disk.nvme0c0n1.out‚3750.55810546875# proc.memory.rssMB‚ 2678.11328125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1659.6953125 disk.dm-1.in‚ 0.10546875 disk.sda.in‚ 0.10546875$ memory_percent‚98.71113300323486éú†É‘Š ‚(:† â¡íÒª¦ÔŠ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚182.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PSMÞ‹ ‚(:Ó â¡íÒï©Ò! disk.nvme0n1.in‚ 1660.17578125% disk.nvme0n1.out‚3758.19970703125 disk.rbd0.in‚0 disk.dm-0.in‚ 1660.17578125 network.recv‚454543 disk.sda3.in‚ 0.10546875( proc.memory.availableMB‚ 210.77734375) disk./.usagePercent‚39.61044905400915 disk.drbd2165.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1660.17578125' disk.nvme0c0n1.out‚3758.19970703125 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.sent‚ 3.566672e+06$ disk./.usageGB‚590.2243156433105 disk.sda3.out‚ 30.94921875 disk.sda.out‚ 30.94921875 disk.rbd1.out‚3.25 disk.sda2.in‚0 disk.dm-1.out‚ 30.94921875$ memory_percent‚98.71351718902588 disk.sda.in‚ 0.10546875 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875# proc.memory.rssMB‚ 2678.11328125* proc.memory.percent‚16.345906257629395 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0" disk.dm-0.out‚3758.19970703125 cpu‚26.778195983128125w(š`2Œ - _wandbruntime‚1326  _runtime‚1326¨RÅ ¿ _step‚223 % lr-AdamW/pg1‚0.000842649754002031 % lr-AdamW/pg2‚0.000842649754002031 trainer/global_step‚2799 % _timestamp‚1.7843693855095532e+09 _runtime‚1326.953359734vJ£ÖÊŽ Ä trainer/global_step‚2799 % _timestamp‚1.7843693855095532e+09 _runtime‚1326.953359734 _step‚223 % lr-AdamW/pg1‚0.000842649754002031 % lr-AdamW/pg2‚0.000842649754002031ß_*2 - _wandbruntime‚1326  _runtime‚1326ržÖÎ' ÈO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108448199281152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108448199281152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6343787 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1701225196032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26517848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102212513753600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106642410938528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47444179 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚209 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100511313461248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19214175 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1639948630176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106802344677376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645837564416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1645749627392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106642514114978 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20926333 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106802344677376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843693855107222e+09 _runtime‚1326.954714311 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108448094304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19213963 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚2799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20926331 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20926331 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6343578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105002462308352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚212 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1640034767266 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1701399403008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645837564416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1645749627392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12870388 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12870385 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47454158 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108448094304768 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102212712864256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100511288557568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106802361716736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106802361716736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20926333 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105002479347712bPY ‘ ÍO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20926331 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20926331 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6343578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105002462308352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚212 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1640034767266 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1701399403008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645837564416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1645749627392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12870388 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12870385 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47454158 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108448094304768 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102212712864256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100511288557568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106802361716736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106802361716736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20926333 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105002479347712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108448199281152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108448199281152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6343787 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1701225196032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26517848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102212513753600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106642410938528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47444179 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚209 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100511313461248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19214175 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1639948630176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106802344677376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645837564416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1645749627392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106642514114978 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20926333 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106802344677376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843693855107222e+09 _runtime‚1326.954714311 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108448094304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19213963 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚2799à,|Þ2’ - _wandbruntime‚1327  _runtime‚1327D²­Ø&“ ÒM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19220606 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106679553471948 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47470674 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20933554 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106839472680960 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6345737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100546200072192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108485774628352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701989848576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 % _timestamp‚1.7843693858468564e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _step‚225 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20933552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26527129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1640602088908 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106839472680960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102248214824448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701813551616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1646406930944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106839455641600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26537120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108485879611904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1646406930944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20933554 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1646318986752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106839455641600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19220840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105038934343680 _runtime‚1327.290910308 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20933552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1646318986752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1640515951706 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100546224975872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47460681 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26527129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108485879611904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26537120 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47470674 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12874869 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6345968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146ôT¾öÝ&” ×M NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26537120 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47470674 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12874869 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6345968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19220606 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106679553471948 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47470674 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20933554 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106839472680960 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6345737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100546200072192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108485774628352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701989848576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 % _timestamp‚1.7843693858468564e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _step‚225 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20933552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26527129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1640602088908 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106839472680960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102248214824448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701813551616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1646406930944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106839455641600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26537120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108485879611904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1646406930944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20933554 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1646318986752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106839455641600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19220840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105038934343680 _runtime‚1327.290910308 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20933552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1646318986752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1640515951706 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100546224975872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47460681 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26527129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108485879611904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192á×½úÄ2• - _wandbruntime‚1327  _runtime‚1327aµâ†– ‚ %loss/total_loss‚9.343106292715204 _step‚226 1loss/SRqqt2/detection_loss‚0.6918622432084713 trainer/global_step‚2799 % _timestamp‚1.7843693858485882e+09 1loss/SRqqt1/detection_loss‚1.0566711474999062 epoch‚3 _runtime‚1327.292117086 /loss/FRt2/assignment_loss‚1.149945669781928 2loss/SRqqt2/assignment_loss‚0.5956279755880862 2loss/FBt1/assignment_loss‚0.006810888096684461 2loss/FBt2/assignment_loss‚0.003342087886243338 /loss/FRt1/detection_loss‚0.6047004961996196 /loss/FRt2/detection_loss‚1.3865357582767805 /loss/FBt2/detection_loss‚0.7712643340907314 0loss/FRt1/assignment_loss‚1.9979711671449027 2loss/SRqqt1/assignment_loss‚0.6495154766849333 /loss/FBt1/detection_loss‚0.42885904825691671- — ‡ 1loss/SRqqt1/detection_loss‚1.0566711474999062 epoch‚3 _runtime‚1327.292117086 /loss/FRt2/assignment_loss‚1.149945669781928 2loss/SRqqt2/assignment_loss‚0.5956279755880862 2loss/FBt1/assignment_loss‚0.006810888096684461 2loss/FBt2/assignment_loss‚0.003342087886243338 /loss/FRt1/detection_loss‚0.6047004961996196 /loss/FRt2/detection_loss‚1.3865357582767805 /loss/FBt2/detection_loss‚0.7712643340907314 0loss/FRt1/assignment_loss‚1.9979711671449027 2loss/SRqqt1/assignment_loss‚0.6495154766849333 /loss/FBt1/detection_loss‚0.4288590482569167 %loss/total_loss‚9.343106292715204 _step‚226 1loss/SRqqt2/detection_loss‚0.6918622432084713 trainer/global_step‚2799 % _timestamp‚1.7843693858485882e+09âs›˜ ‚(: ñ¡íÒØ•ÍŒ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚171.312' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.26240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ßÕ™ ‚(:Ê ñ¡íÒÝûô disk.dm-0.in‚ 1660.17578125 disk.sda1.in‚0$ memory_percent‚99.10387992858887" proc.memory.rssMB‚ 2678.1171875 network.sent‚ 3.571664e+06 disk.sda2.out‚0 disk.sda3.out‚ 31.328125' proc.memory.availableMB‚ 146.8203125 disk.dm-1.in‚ 0.10546875# disk.nvme0c0n1.in‚ 1660.17578125' disk.nvme0c0n1.out‚3761.40283203125 network.recv‚457937 disk.sda.out‚ 31.328125 disk.rbd1.out‚3.25 cpu‚26.78644931646801! disk.nvme0n1.in‚ 1660.17578125% disk.nvme0n1.out‚3761.40283203125 disk.dm-1.out‚ 31.328125 disk.sda3.in‚ 0.10546875$ disk./.usageGB‚590.2229385375977 disk.drbd2165.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.dm-0.out‚3761.40283203125 disk.sda.in‚ 0.10546875 disk.drbd2165.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.345930099487305) disk./.usagePercent‚39.61035663528929 disk.sda1.out‚0 disk.sda2.in‚0ÕP!2š - _wandbruntime‚1349  _runtime‚1349[¥ƒÇ› Á & lr-AdamW/pg1‚0.0008423761606335263 & lr-AdamW/pg2‚0.0008423761606335263 trainer/global_step‚2849 % _timestamp‚1.7843694076045291e+09 _runtime‚1349.048246968 _step‚227"ó?&Ìœ Æ % _timestamp‚1.7843694076045291e+09 _runtime‚1349.048246968 _step‚227 & lr-AdamW/pg1‚0.0008423761606335263 & lr-AdamW/pg2‚0.0008423761606335263 trainer/global_step‚2849ãé:;2 - _wandbruntime‚1349  _runtime‚1349H…È,?ž ÇO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48271131 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1674249852416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103987704427008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106826064074752 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108657892888576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 1349.04956208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19546637 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108494513861086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103987503219200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108494410684636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106826081114112 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13094588 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110332142740992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21287383 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.784365»¾¤Ž94076055343e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1668346609884 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108657892888576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚228 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110332142740992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26993727 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102256889192448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21287381 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19546848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108657909927936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1730638930432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110332247717376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21287383 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1674337789440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6452052 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102256864288768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1668432746974 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1674337789440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26993727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108657909927936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1668432746974 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1674337789440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26993727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108657909927936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19546637 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108494513861086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103987503219200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108494410684636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106826081114112 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13094588 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110332142740992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21287383 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843694076055343e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1668346609884 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108657892888576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚228 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110332142740992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26993727 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102256889192448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21287381 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19546848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108657909927936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1730638930432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146ä­‚Ç72  - _wandbruntime‚1349  _runtime‚1349¶nIœØ&¡ ÒM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106862536110080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104023206157824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668913931414 % _timestamp‚1.7843694079427488e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110369928048128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102291800707072 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108695003852800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27003022 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21294604 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1731405450752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚2849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13099069 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21294602 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19553485 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48297626 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48287633 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110369928048128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108531450041494 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108695020892160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21294604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13099072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1731229153792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _runtime‚1349.386820214 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110369823064576 _step‚229 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26993031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108695020892160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106862553149440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19553255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104023004957184 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2©¹s¢ ×M XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1674819211776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108695003852800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27003022 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21294604 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1731405450752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚2849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13099069 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21294602 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19553485 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48297626 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48287633 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110369928048128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108531450041494 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108695020892160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21294604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13099072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1731229153792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _runtime‚1349.386820214 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110369823064576 _step‚229 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26993031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108695020892160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106862553149440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 K>DeviceStatsMo¶;È÷jnitor.on_train_batch_end/inactive_split.all.freed‚19553255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104023004957184 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106862536110080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104023206157824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668913931414 % _timestamp‚1.7843694079427488e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110369928048128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102291800707072 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112216191177216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108649665841152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104002464923648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚110513441099776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27449652 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696744589592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104002440019968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21648433 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6561025 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112216296153600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1760098163200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚110513458139136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6560795 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49108062 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105762538183168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚110513441099776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19879580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19879813 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13318785 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1696830726682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚1371.328669633 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21648433 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27459629 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112216296153600 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21648431 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327_õn=Ó'¯ ÍO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1696830726682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚1371.328669633 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21648433 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27459629 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112216296153600 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21648431 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49098083 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13318788 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105762735196672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112216191177216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108649665841152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104002464923648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚110513441099776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27449652 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696744589592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104002440019968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21648433 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6561025 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112216296153600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1760098163200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚110513458139136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6560795 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49108062 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105762538183168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚110513441099776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19879580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19879813 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13318785 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0èpÑKl2° - _wandbruntime‚1371  _runtime‚13717˜¨ã> ± ÒM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13323269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104037376438272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _step‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49124578 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105798033601024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108686154915840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1703319436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112253871500800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1703407380992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112253976484352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21655652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6563163 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110383449787602 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110550569103360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27468924 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27458933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13323272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108686137876480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21655654 @:DeviceStatsMonitor.on_train_batch_end/nuî-[©šm_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49114585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110550552064000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 trainer/global_step‚2899 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49124578 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21655654 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1697311911122 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110383552964164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1703319436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1703407380992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112253976484352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19886203 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1760860460544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1697398048324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473^š®ÔÝ&² ×M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19886435 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110550569103360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21655652 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27458933 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110550552064000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49114585 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104037351534592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1703407380992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112253976484352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19886203 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1760860460544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1697398048324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13323269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104037376438272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _step‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49124578 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105798033601024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108686154915840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1703319436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112253871500800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1703407380992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112253976484352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21655652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6563163 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110383449787602 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110550569103360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27468924 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27458933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13323272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108686137876480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21655654 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49114585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110550552064000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 trainer/global_step‚2899 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49124578 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21655654 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1697311911122 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110383552964164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1703319436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105748040654848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚2949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112368989310976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1731338239488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20213200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1789534630400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1731250302464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1725228706390 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13542988 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110473267607552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27915554 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13542985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110473284646912 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112369006350336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49935014 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚236 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114100344589824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114100344589824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27915554 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1731338239488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1789706740224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6670215 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105748015751168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22009481 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114100239613440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6670441 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114100239613440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107537747395072 _runtime‚1393.423664877 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107537550381568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143¾"Ͻ ½ ÍO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112198410176852 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105748040654848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚2949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112368989310976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1731338239488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20213200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1789534630400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1731250302464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 S»êÐEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1725228706390 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13542988 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110473267607552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27915554 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13542985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110473284646912 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112369006350336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49935014 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚236 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114100344589824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114100344589824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27915554 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1731338239488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1789706740224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6670215 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105748015751168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22009481 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114100239613440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6670441 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114100239613440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107537747395072 _runtime‚1393.423664877 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107537550381568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112368989310976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1725142569300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ìd12¾ - _wandbruntime‚1393  _runtime‚1393#Ä· ×&¿ ÑM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114138024920576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49951530 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13547472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22016702 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22016702 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112406100275200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1725796028032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6672298 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105782952169472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22016704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112406117314560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49941537 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110509756682240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27924835 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27934826 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112235449533710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1731819661824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1731907606016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13547469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20219984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1393.760739376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105782927265792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49941537 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107573045881344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1790118615552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚237 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107573249179136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790297009664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22016704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1725709890830 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112406117314560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20219767 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112406100275200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6672512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27924835 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112235552710272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49951530 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114137919937024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473”ëkXÜ&À ÖM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114137919937024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114138024920576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49951530 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13547472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22016702 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22016702 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112406100275200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1725796028032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6672298 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105782952169472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22016704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112406117314560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49941537 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110509756682240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27924835 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27934826 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112235449533710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1731819661824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1731907606016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13547469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20219984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1393.760739376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105782927265792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49941537 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107573045881344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1790118615552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚237 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107573249179136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790297009664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22016704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1725709890830 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112406117314560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20219767 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112406100275200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6672512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27924835 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112235552710272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49951530 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993í ÑB÷2Á - _wandbruntime‚1393  _runtime‚1393£ä· € 0loss/FRt1/assignment_loss‚1.9006201511621474 /loss/FRt2/detection_loss‚1.3175699546152506 /loss/FBt1/detection_loss‚0.9087338583736584 trainer/global_step‚2949 0loss/FRt2/assignment_loss‚1.0148223732931043 /loss/FRt1/detection_loss‚0.6463307237926347 1loss/SRqqt2/detection_loss‚0.6667608490226686 $loss/total_loss‚9.38951276407142 _runtime‚ 1393.76216123 2loss/FBt2/assignment_loss‚0.006512064859308846 % _timestamp‚1.7843694523186371e+09 2loss/SRqqt1/assignment_loss‚0.6489952978135335 2loss/SRqqt2/assignment_loss‚0.5141100586736381 1loss/FBt1/assignment_loss‚0.06282150808655435 1loss/SRqqt1/detection_loss‚0.9498169895248125 /loss/FBt2/detection_loss‚0.7524189348541089 epoch‚3 _step‚238z§ à … 0loss/FRt1/assignment_loss‚1.9006201511621474 /loss/FRt2/detection_loss‚1.3175699546152506 /loss/FBt1/detection_loss‚0.9087338583736584 trainer/global_step‚2949 0loss/FRt2/assignment_loss‚1.0148223732931043 /loss/FRt1/detection_loss‚0.6463307237926347 1loss/SRqqt2/detection_loss‚0.6667608490226686 $loss/total_loss‚9.38951276407142 _runtime‚ 1393.76216123 2loss/FBt2/assignment_loss‚0.006512064859308846 % _timestamp‚1.7843694523186371e+09 2loss/SRqqt1/assignment_loss‚0.6489952978135335 2loss/SRqqt2/assignment_loss‚0.5141100586736381 1loss/FBt1/assignment_loss‚0.06282150808655435 1loss/SRqqt1/detection_loss‚0.9498169895248125 /loss/FBt2/detection_loss‚0.7524189348541089 epoch‚3 _step‚238î+ÓŽdÄ ‚(:… ­¢íÒ‹Ç‹ gpu.0.gpu‚66.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚184.225' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06 ' ÝÅ ‚(:Ò ­¢íÒýîÝŽ% disk.nvme0n1.out‚3778.64208984375 disk.sda.in‚ 0.10546875 network.recv‚477389 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚3778.64208984375 disk.sda3.in‚ 0.10546875 disk.sda3.out‚ 32.13671875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1737.6328125 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875 network.sent‚ 3.777364e+06$ disk./.usageGB‚590.2232933044434" disk.nvme0c0n1.in‚ 1737.6328125 disk.drbd2165.out‚0 disk.sda.out‚ 32.13671875 disk.drbd2165.in‚0* proc.memory.percent‚16.345930099487305 cpu‚26.812644193419196 disk.rbd1.out‚3.4375 disk.sda2.in‚0$ memory_percent‚99.10733699798584( proc.memory.availableMB‚ 146.25390625" proc.memory.rssMB‚ 2678.1171875 disk.sda2.out‚0 disk.dm-1.out‚ 32.13671875 disk.dm-0.in‚ 1737.6328125* disk./.usagePercent‚39.610380443989975' disk.nvme0c0n1.out‚3778.64208984375ðDÒ‘Æ ‚(:† ¼¢íÒÕ‹ÆŠ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚180.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÙÇ ‚(:Î ¼¢íÒè‡èŒ disk.drbd2165.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚3785.10205078125 network.recv‚482797 disk.nvme0n1.in‚ 1737.6328125 disk.dm-1.in‚ 0.10546875" disk.dm-0.out‚3785.10205078125' proc.memory.availableMB‚ 145.9921875 disk.sda.out‚ 32.50390625 disk.dm-1.out‚ 32.50390625 disk.rbd0.out‚0* disk./.usagePercent‚39.610398620524904$ disk./.usageGB‚590.2235641479492 disk.sda3.in‚ 0.10546875 disk.sda.in‚ 0.10546875 disk.rbd1.out‚3.5 network.sent‚ 3.827226e+06$ memory_percent‚99.10893440246582 disk.rbd1.in‚0" proc.memory.rssMB‚ 2678.1171875 cpu‚26.818846864132816 proc.cpu.threads‚17 disk.sda3.out‚ 32.50390625" disk.nvme0c0n1.in‚ 1737.6328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1737.6328125* proc.memory.percent‚16.345930099487305 disk.drbd2165.out‚0% disk.nvme0n1.out‚3785.10205078125 disk.rbd0.in‚0Z92È - _wandbruntime‚1415  _runtime‚1415,æ&ÐÇÉ Á & lr-AdamW/pg1‚0.0008415151946502841 & lr-AdamW/pg2‚0.0008415151946502841 trainer/global_step‚2999 % _timestamp‚1.7843694742965205e+09 _runtime‚1415.740301306 _step‚239ëâ×-ÌÊ Æ & lr-AdamW/pg1‚0.0008415151946502841 & lr-AdamW/pg2‚0.0008415151946502841 trainer/global_step‚2999 % _timestamp‚1.7843694742965205e+09 _runtime‚1415.740301306 _step‚239ï.w™s2Ë - _wandbruntime‚1415  _runtime‚1415`«{&„Ì ÈO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107493591482368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6779087 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114050513099410 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109312778465280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114050409922960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759750527488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 % _timestamp‚1.7843694742976303e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1819162079232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114224554561536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50761966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107493616386048 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115984288049664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115984393026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114224554561536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112296886413312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114224537522176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115984288049664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1759838464512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109312568868864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1753626686098 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1753540549008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20546036 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20546275 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22370533 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28391433 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13767188 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115984393026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114224537522176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28391433 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22370531 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28381456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22370531 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0qêÖÓ'Í ÍO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22370533 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28391433 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13767188 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115984393026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114224537522176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28391433 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22370531 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28381456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22370531 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107493591482368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6779087 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114050513099410 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109312778465280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114050409922960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759750527488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 % _timestamp‚1.7843694742976303e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1819162079232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114224554561536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50761966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107493616386048 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115984288049664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115984393026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114224554561536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112296886413312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114224537522176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115984288049664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1759838464512 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109312568868864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1753626686098 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1753540549008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20546036 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20546275ðÂò’š2Î - _wandbruntime‚1416  _runtime‚1416¯;QzØ&Ï ÒM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50768489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22377752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114261648486400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114261665525760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6781007 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114087449279818 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20552676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116022073356800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22377754 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22377752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28390737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13771672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1754107870538 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50778482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107528502996992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22377754 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6781244 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _step‚241 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50778482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1760407831040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109348270201856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28390737 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1760319886848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1819742301184 _runtime‚1416.081650222 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1760407831040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107528527900672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640®Ñж Ð ×M EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20552676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116022073356800 [MDeviceStatsMonitor.on_train_-‹Ü¾'batch_end/inactive_split_bytes.small_pool.current‚ 180491264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22377754 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22377752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28390737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13771672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1754107870538 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50778482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107528502996992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22377754 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6781244 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _step‚241 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50778482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1760407831040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109348270201856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28390737 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1760319886848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1819742301184 _runtime‚1416.081650222 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1760407831040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107528527900672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50768489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22377752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114261648486400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114261665525760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6781007 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114087449279818 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117868336485888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20878809 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22731583 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51588918 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20878588 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22731581 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116080085733376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22731581 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28857335 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117868441462272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28857335 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109239192117248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1788338689536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843694964756603e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51578939 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114120488179712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚244 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6887421 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116080102772736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116080085733376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13991388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1437.919671861 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117868336485888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13991385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111087549088768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0IÂ4 Ó'Û ÍO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1788338689536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843694964756603e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51578939 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114120488179712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚244 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6887421 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116080102772736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116080085733376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13991388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1437.919671861 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117868336485888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13991385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111087549088768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117868441462272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22731583 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6887203 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116080102772736 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111087744005120 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117868336485888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20878809 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22731583 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51588918 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20878588 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22731581 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116080085733376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22731581 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28857335 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117868441462272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28857335 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109239192117248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ôGs^ï2Ü - _wandbruntime‚1438  _runtime‚1438§ìÝ ÒM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115939449025926 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114156943175680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28856639 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1788820111872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1788820111872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K¯ódÆ!>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20885230 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22738802 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚3049 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 _runtime‚1438.256565035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115939552202488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22738804 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111123053342720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51595441 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22738802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51595441 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1848974614528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111123246154752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109274078728192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114156960215040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚215 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13995872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1782591987448 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13995869 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6889361 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28856639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6889573 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28866630 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116117196697600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116117196697600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51605434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117906121793024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22738804 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1788908056064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 68451041287PÃÝ&Þ ×M LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1788820111872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1788820111872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20885230 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22738802 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚3049 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 _runtime‚1438.256565035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115939552202488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22738804 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111123053342720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51595441 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22738802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51595441 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1848974614528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111123246154752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109274078728192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114156960215040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚215 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13995872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1782591987448 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13995869 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6889361 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28856639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6889573 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28866630 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116117196697600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116117196697600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51605434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117906121793024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22738804 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1788908056064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115939449025926 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114156943175680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28856639 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0õ3Ç¥2ß - _wandbruntime‚1438  _runtime‚1438-á$ à  2loss/FBt1/assignment_loss‚0.007043393661533533 1loss/SRqqt1/detection_loss‚0.9680317091670904 epoch‚3 0loss/FRt2/assignment_loss‚1.2487148227113667 1loss/SRqqt2/detection_loss‚0.6979572116722201 /loss/FBt1/detection_loss‚0.6977543801069259 _runtime‚1438.258006807 _step‚246 2loss/SRqqt1/assignment_loss‚0.7473113122689244 /loss/FRt1/detection_loss‚0.5930776360001723 /loss/FBt2/detection_loss‚0.6777662831462092 %loss/total_loss‚9.887875455889716 0loss/FRt1/assignment_loss‚2.0823752782201073 2loss/SRqqt2/assignment_loss‚0.6289299378851104 1loss/FBt2/assignment_loss‚0.04985938409100004 .loss/FRt2/detection_loss‚1.489054106959057 trainer/global_step‚3049 % _timestamp‚1.7843694968144708e+09}ÜÌs á † 0loss/FRt2/assignment_loss‚1.2487148227113667 1loss/SRqqt2/detection_loss‚0.6979572116722201 /loss/FBt1/detection_loss‚0.6977543801069259 _runtime‚1438.258006807 _step‚246 2loss/SRqqt1/assignment_loss‚0.7473113122689244 /loss/FRt1/detection_loss‚0.5930776360001723 /loss/FBt2/detection_loss‚0.6777662831462092 %loss/total_loss‚9.887875455889716 0loss/FRt1/assignment_loss‚2.0823752782201073 2loss/SRqqt2/assignment_loss‚0.6289299378851104 1loss/FBt2/assignment_loss‚0.04985938409100004 .loss/FRt2/detection_loss‚1.489054106959057 trainer/global_step‚3049 % _timestamp‚1.7843694968144708e+09 2loss/FBt1/assignment_loss‚0.007043393661533533 1loss/SRqqt1/detection_loss‚0.9680317091670904 epoch‚3öþqvœâ ‚(:‘ Ú¢íÒ ÁŠ gpu.0.gpu‚67.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I¦ÕuÖã ‚(:Ë Ú¢íÒè•éŒ disk.dm-1.in‚ 0.10546875 network.sent‚ 3.894622e+06 disk.sda3.out‚ 32.86328125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 32.86328125) disk./.usagePercent‚39.61042089318039 disk.sda.in‚ 0.10546875 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1816.70703125* proc.memory.percent‚16.345930099487305 disk.drbd2165.in‚0 cpu‚26.830555179365255 disk.dm-0.out‚ 3793.58203125 disk.sda.out‚ 32.86328125 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚590.2238960266113 disk.sda3.in‚ 0.10546875# disk.nvme0c0n1.in‚ 1816.70703125" proc.memory.rssMB‚ 2678.1171875$ disk.nvme0c0n1.out‚ 3793.58203125$ memory_percent‚99.10762310028076( proc.memory.availableMB‚ 146.20703125 network.recv‚491049 disk.rbd1.out‚3.5625! disk.nvme0n1.in‚ 1816.70703125" disk.nvme0n1.out‚ 3793.58203125otÛ‘ä ‚(:† é¢íÒÀþØ‹ gpu.0.gpu‚67.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚115.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ždrœÛå ‚(:Ð é¢íÒ½ö† disk.sda1.out‚0$ memory_percent‚99.10697937011719 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚3796.07861328125 disk.dm-1.in‚ 0.10546875 disk.sda.in‚ 0.10546875! disk.nvme0n1.in‚ 1816.70703125$ proc.memory.availableMB‚146.3125* proc.memory.percent‚16.345930099487305 cpu‚26.83647605600394 disk.sda3.out‚ 32.94921875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1816.70703125' disk.nvme0c0n1.out‚3796.07861328125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2678.1171875* disk./.usagePercent‚39.610432669526965 disk.rbd1.out‚3.5625 disk.dm-1.out‚ 32.94921875$ disk./.usageGB‚590.2240715026855 disk.rbd0.in‚0 network.sent‚ 3.956564e+06 network.recv‚496350 disk.dm-0.in‚ 1816.70703125" disk.dm-0.out‚3796.07861328125 disk.sda3.in‚ 0.10546875 proc.cpu.threads‚17 disk.sda.out‚ 32.94921875 disk.rbd1.in‚0hÎK2æ - _wandbruntime‚1459  _runtime‚1459Š‘,mÅç ¿ $ _timestamp‚1.784369518438509e+09 _runtime‚ 1459.88227457 _step‚247 & lr-AdamW/pg1‚0.0008409077753316755 & lr-AdamW/pg2‚0.0008409077753316755 trainer/global_step‚3099¿Û GÊè Ä & lr-AdamW/pg1‚0.0008409077753316755 & lr-AdamW/pg2‚0.0008409077753316755 trainer/global_step‚3099 $ _timestamp‚1.784369518438509e+09 _runtime‚ 1459.88227457 _step‚247÷!Cð2é - _wandbruntime‚1459  _runtime‚1459çIåå#ê ÈO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117935650983936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112862800141824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112862603128320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117754512591626 % _timestamp‚1.7843695184394865e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29313260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21211444 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23092633 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21211207 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117935650983936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117935633944576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110984767848448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14215588 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6995622 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117754409415176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1816838914560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23092633 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14215585 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1878032293376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1810336508424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1816838914560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119752384922112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1810422645514 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6995856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110984742944768 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119752489898496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29323237 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29323237 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1816750977536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52415870 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115944089946112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23092631 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1877860183552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115944072906752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23092631 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119752384922112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119752489898496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119752384922112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119752489898496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117754512591626 % _timestamp‚1.7843695184394865e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29313260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21211444 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23092633 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21211207 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117935650983936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117935633944576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110984767848448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14215588 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6995622 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117754409415176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1816838914560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23092633 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14215585 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1878032293376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1810336508424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1816838914560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119752384922112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1810422645514 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6995856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110984742944768 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119752489898496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29323237 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29323237 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1816750977536øa<»þ2ì - _wandbruntime‚1460  _runtime‚1460î6H×&í ÑM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1817408281088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117791448772034 _runtime‚ 1460.21948051 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23099854 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117791551948596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52422393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117972744908800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1878445159936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117972761948160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6997791 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14220072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23099852 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6998013 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1817320336896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117972761948160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21217860 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112898099619328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1817320336896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111019654459392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1878623554048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1810989967156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117972744908800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1810903829954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23099852 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119790170229248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52432386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52422393 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29332532 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552¤™ýÜ&î ÖM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 % _timestamp‚1.7843695187753856e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14220069 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29322541 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1817408281088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115980544942080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119790065245696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29322541 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111019679363072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52432386 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117791551948596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52422393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117972744908800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1878445159936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117972761948160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6997791 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14220072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23099852 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6998013 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1817320336896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117972761948160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21217860 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112898099619328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1817320336896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111019654459392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1878623554048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1810989967156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117972744908800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1810903829954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23099852 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119790170229248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52432386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52422393 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29332532 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1817408281088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117791448772034 _runtime‚ 1460.21948051 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23099854 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23453681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121636433358336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119791199195136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845251202560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114637651274752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119606512337734 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1838734488132 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29779162 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _step‚252 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29779162 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119791182155776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1838820625222 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53242822 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53232843 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119791199195136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1845339139584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1907502611456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119606409161284 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21544717 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21544485 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114637846191104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29789139 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23453681 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112730343579648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14439788 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843695406779304e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845251202560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23453683 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7104700 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Ò'ù ÌO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121636433358336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _runtime‚ 1482.12193349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1907332598784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121636433358336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119791199195136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845251202560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114637651274752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119606512337734 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1838734488132 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29779162 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _step‚252 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29779162 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119791182155776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1838820625222 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53242822 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53232843 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119791199195136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1845339139584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1907502611456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119606409161284 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21544717 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21544485 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114637846191104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29789139 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23453681 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112730343579648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14439788 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843695406779304e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845251202560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23453683 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7104700 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23453681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23460902 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1839301809662 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843695410139809e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21551358 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14444272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29798434 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119643448518142 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117804163747840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121674218665472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚253 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117804146708480 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121674113681920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1845908506112 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53259338 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7106854 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112765230190592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23460902 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 _runtime‚1482.458009238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1845820561920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119828310159360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14444269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53249345 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53249345 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1845908506112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1908098340864 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21551123 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7107086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121674218665472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53259338 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1907922043904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29788443 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29788443 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119828310159360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53259338 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 QEDeviceSta¬ª›ñj%tsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7106854 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112765230190592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23460902 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 _runtime‚1482.458009238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1845820561920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119828310159360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14444269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53249345 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53249345 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1845908506112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1908098340864 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21551123 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7107086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121674218665472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53259338 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1907922043904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29788443 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29788443 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119828310159360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23460902 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1839301809662 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843695410139809e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21551358 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14444272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29798434 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119643448518142 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117804163747840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121674218665472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚253 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117804146708480 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121674113681920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1845908506112 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123520481794560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54069774 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1867218604930 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121646747406336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30255041 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21877342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7213586 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116412871845888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123520586770944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1867132467840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1873839364608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123520586770944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14663988 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1873751427584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14663985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114475894407168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1873751427584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1936780425216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121458512083842 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121646747406336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213357 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114475919310848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30245064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21877574 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128wFß]‰ ÍO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1873839364608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123520586770944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14663988 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1873751427584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14663985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114475894407168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1873751427584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1936780425216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121458512083842 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121646747406336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213357 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114475919310848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30245064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21877574 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121646730366976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116412674832384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123520481794560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54069774 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pooPÖ—Åvl.allocated‚ 1867218604930 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121646747406336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30255041 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21877342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7213586 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116412871845888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123520586770944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1867132467840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0€æóÁ2Š - _wandbruntime‚1504  _runtime‚1504gðÔØ&‹ ÒM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121683841331200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7215769 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23821952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21884003 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116448171958272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23821954 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30264336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114510805921792 trainer/global_step‚3199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30254345 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7215534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1874408731136 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1504.483944657 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121683841331200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54086290 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116448375256064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119627765514240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30264336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123558267101696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30254345 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54086290 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54076297 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54076297 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1874320786944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21884241 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1867785926572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23821952 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473LÙå÷Ý&Œ ×M EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1504.483944657 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121683841331200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54086290 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116448375256064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119627765514240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30264336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123558267101696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30254345 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54086290 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54076297 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54076297 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1874320786944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21884241 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1867785926572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23821952 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121683841331200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7215769 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23821952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21884003 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116448171958272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23821954 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30264336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114510805921792 trainer/global_step‚3199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30254345 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7215534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1874408731136Ó™Û2 - _wandbruntime‚1504  _runtime‚1504Ç¢?"Ž ‚ 2loss/FBt2/assignment_loss‚0.033787493305582394 _runtime‚1504.484834577 /loss/FRt1/detection_loss‚0.5624718302774099 /loss/FRt2/detection_loss‚1.2482988562675075 1loss/SRqqt2/detection_loss‚0.7055834018524856 $loss/total_loss‚9.72316319971731 0loss/FRt2/assignment_loss‚1.1610229700569092 2loss/SRqqt1/assignment_loss‚0.7422650791377735 2loss/SRqqt2/assignment_loss‚0.6416157872320027 1loss/SRqqt1/detection_loss‚1.0760013642146775 trainer/global_step‚3199 $ _timestamp‚1.784369563041156e+09 _step‚258 0loss/FRt1/assignment_loss‚1.9028836053340032 3loss/FBt1/assignment_loss‚0.0004075426209476281 /loss/FBt1/detection_loss‚1.1342386584728956 /loss/FBt2/detection_loss‚0.5145866109451163 epoch‚3—¯ò  ‡ /loss/FBt1/detection_loss‚1.1342386584728956 /loss/FBt2/detection_loss‚0.5145866109451163 epoch‚3 2loss/FBt2/assignment_loss‚0.033787493305582394 _runtime‚1504.484834577 /loss/FRt1/detection_loss‚0.5624718302774099 /loss/FRt2/detection_loss‚1.2482988562675075 1loss/SRqqt2/detection_loss‚0.7055834018524856 $loss/total_loss‚9.72316319971731 0loss/FRt2/assignment_loss‚1.1610229700569092 2loss/SRqqt1/assignment_loss‚0.7422650791377735 2loss/SRqqt2/assignment_loss‚0.6416157872320027 1loss/SRqqt1/detection_loss‚1.0760013642146775 trainer/global_step‚3199 $ _timestamp‚1.784369563041156e+09 _step‚258 0loss/FRt1/assignment_loss‚1.9028836053340032 3loss/FBt1/assignment_loss‚0.0004075426209476281‚Óä"› ‚(: ¥£íÒª“™ gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚148.016' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.603199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E7 áÙ‘ ‚(:Î ¥£íÒè±åŽ* disk./.usagePercent‚39.610432413519426 disk.sda2.out‚0 disk.dm-0.in‚ 1949.234375' disk.nvme0c0n1.out‚3813.17919921875% disk.nvme0n1.out‚3813.17919921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚590.2240676879883 disk.rbd1.out‚3.6875$ memory_percent‚99.12612438201904 network.recv‚513633 disk.sda.in‚ 0.10546875 disk.rbd1.in‚0! proc.memory.rssMB‚ 2681.421875 disk.nvme0n1.in‚ 1949.234375" disk.dm-0.out‚3813.17919921875( proc.memory.availableMB‚ 143.17578125 cpu‚26.884745637299034 disk.sda3.in‚ 0.10546875 disk.sda.out‚ 34.15234375* proc.memory.percent‚16.366100311279297 network.sent‚ 4.104845e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 34.15234375 disk.drbd2165.in‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 1949.234375 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚ 34.15234375Ú×™ 2’ - _wandbruntime‚1526  _runtime‚1526=?ræi“ ¸{ & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6911046504974365 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30035558342933655 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6869152784347534 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4608136713504791 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00545900221914053 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8048534393310547 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 particle/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08608182519674301 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391015291213989 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964994788169861 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03227057307958603 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.014682539738714695 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.057172633707523346 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5257812738418579 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01886453852057457 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 (loss/val_total_loss‚9.30238822529445 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25645431876182556 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1888359636068344 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7353945374488831 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02044961228966713 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07059917598962784 particle/accuracy_1_of_2‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11530325561761856 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.537826418876648 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18548476696014404 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981576800346375 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.411263108253479 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7423811554908752 +jet/accuracy_1_of_1‚0.28263217210769653 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.684249997138977 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6895982027053833 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8595890998840332 +jet/accuracy_3_of_3‚0.07157012075185776 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6240577101707458 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4662972390651703 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9401522874832153 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.381633996963501 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6293729543685913 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.791534423828125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8122806549072266 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5784085392951965 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6890245676040649 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09415973722934723 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8066151142120361 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6789921522140503 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07071764022111893 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6681569218635559 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8424811363220215 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957747459411621 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21505077183246613 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.789872407913208 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09128113836050034 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04462439566850662 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010435301810503006 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10018441826105118 +particle/specificity‚0.9087850451469421 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6093365550041199 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37345680594444275 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7100987434387207 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7888603210449219 /particle/accuracy_3_of_3‚0.9989849328994751 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880256652832031 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 _runtime‚1526.850899017 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10189276933670044 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7913024425506592 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9193439483642578 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8863328099250793 6"Purity/0FRt2SRqqt0FBt/event_purity‚0.4227314889431 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9583911895751953 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *particle/sensitÀ'‰›U ivity‚0.618705153465271 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.547221839427948 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.0885293111205101 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_2_of_2‚0.9998950958251953 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.236111119389534 jet/accuracy_3_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.056645218282938004 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8066151142120361 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09102697670459747 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10520833730697632 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚3203 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.693513035774231 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947029948234558 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9657313823699951 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982757031917572 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.731517493724823 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11567284166812897 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8359648585319519 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6490527391433716 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05825331062078476 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43939390778541565 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08387988805770874 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5387881398200989 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6536213755607605 .particle/accuracy_3_of_4‚0.995192289352417 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.0772542953491211 'particle/f_score‚0.6338031888008118 epoch‚3 _step‚259 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.846493661403656 6validation_average_jet_accuracy‚0.3325282335281372 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6737663149833679 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9236235618591309 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6536213755607605 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7509030699729919 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10713686048984528 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04462439566850662 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_2‚0.758546769618988 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7393161654472351 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997220516204834 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6021150350570679 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08735107630491257 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37937694787979126 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5926506519317627 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_3‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48778456449508667 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8102563619613647 $ _timestamp‚1.784369585406424e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975199103355408 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6447047591209412 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297132968902588 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977059364318848 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +jet/accuracy_2_of_2‚0.13866178691387177 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08505700528621674 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6"Purity/*FRt*SRqqt2FBt/event_purity‚0.8751220703125 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6467421054840088 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 particle/accuracy_2_of_4‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09152011573314667 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945631623268127 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913784861564636 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1150251030921936 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9917399883270264 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09102697670459747 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0030699209310114384 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934656023979187 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879684448242188 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956779479980469 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6494718790054321 *jet/accuracy_2_of_3‚0.5069586634635925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6293172836303711 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7159919142723083 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5969748497009277 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13909205794334412 particle/accuracy_2_of_3‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05743424594402313 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5769627690315247 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10608993470668793 +jet/accuracy_3_of_4‚0.09276147931814194 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1696925014257431 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984037280082703 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977059364318848 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09633272886276245 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44019126892089844 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7350451350212097 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1055312380194664 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094xs]YÃ=” ½{ :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07071764022111893 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6681569218635559 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8424811363220215 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957747459411621 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21505077183246613 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.789872407913208 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09128113836050034 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04462439566850662 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010435301810503006 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10018441826105118 +particle/specificity‚0.9087850451469421 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6093365550041199 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37345680594444275 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7100987434387207 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7888603210449219 /particle/accuracy_3_of_3‚0.9989849328994751 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880256652832031 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 _runtime‚1526.850899017 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10189276933670044 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7913024425506592 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9193439483642578 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8863328099250793 6"Purity/0FRt2SRqqt0FBt/event_purity‚0.4227314889431 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9583911895751953 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *particle/sensitivity‚0.618705153465271 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.547221839427948 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.0885293111205101 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_2_of_2‚0.9998950958251953 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.236111119389534 jet/accuracy_3_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.056645218282938004 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8066151142120361 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09102697670459747 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10520833730697632 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚3203 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.693513035774231 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947029948234558 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9657313823699951 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982757031917572 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.731517493724823 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11567284166812897 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8359648585319519 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6490527391433716 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05825331062078476 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43939390778541565 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08387988805770874 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5387881398200989 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6536213755607605 .particle/accuracy_3_of_4‚0.995192289352417 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.0772542953491211 'particle/f_score‚0.6338031888008118 epoch‚3 _step‚259 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.846493661403656 6validation_average_jet_accuracy‚0.3325282335281372 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6737663149833679 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9236235618591309 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6536213755607605 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7509030699729919 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10713686048984528 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04462439566850662 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_2‚0.758546769618988 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7393161654472351 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997220516204834 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6021150350570679 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08735107630491257 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37937694787979126 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5926506519317627 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_3‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48778456449508667 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8102563619613647 $ _timestamp‚1.784369585406424e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975199103355408 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6447047591209412 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297132968902588 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977059364318848 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +jet/accuracy_2_of_2‚0.13866178691387177 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08505700528621674 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6"Purity/*FRt*SRqqt2FBt/event_purity‚0.8751220703125 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6467421054840088 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 particle/accuracy_2_of_4‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09152011573314667 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945631623268127 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913784861564636 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1150251030921936 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9917399883270264 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09102697670459747 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0030699209310114384 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934656023979187 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879684448242188 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956779479980469 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6494718790054321 *jet/accuracy_2_of_3‚0.5069586634635925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6293172836303711 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7159919142723083 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5969748497009277 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13909205794334412 particle/accuracy_2_of_3‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05743424594402313 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5769627690315247 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10608993470668793 +jet/accuracy_3_of_4‚0.09276147931814194 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1696925014257431 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984037280082703 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977059364318848 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09633272886276245 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44019126892089844 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7350451350212097 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1055312380194664 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6911046504974365 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30035558342933655 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6869152784347534 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4608136713504791 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00545900221914053 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8048534393310547 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 particle/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08608182519674301 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391015291213989 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964994788169861 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03227057307958603 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.014682539738714695 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.057172633707523346 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5257812738418579 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01886453852057457 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 (loss/val_total_loss‚9.30238822529445 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25645431876182556 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1888359636068344 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7353945374488831 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02044961228966713 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07059917598962784 particle/accuracy_1_of_2‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11530325561761856 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.537826418876648 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18548476696014404 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981576800346375 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.411263108253479 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7423811554908752 +jet/accuracy_1_of_1‚0.28263217210769653 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.684249997138977 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6895982027053833 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8595890998840332 +jet/accuracy_3_of_3‚0.07157012075185776 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6240577101707458 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4662972390651703 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9401522874832153 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.381633996963501 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6293729543685913 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.791534423828125 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8122806549072266 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5784085392951965 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6890245676040649 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09415973722934723 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8066151142120361 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6789921522140503 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1ƒñ©¹n• ‚(:… ´£íÒ€†Á‹ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VˆeØ– ‚(:Í ´£íÒÆäý network.recv‚518302 disk.dm-1.out‚ 34.45703125* proc.memory.percent‚16.425251960754395 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 1949.234375$ memory_percent‚98.78952503204346 cpu‚26.919428258379863 disk.rbd1.in‚0' disk.nvme0c0n1.out‚3879.75244140625 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2691.11328125 proc.cpu.threads‚17 disk.sda.out‚ 34.45703125 disk.rbd1.out‚3.75 disk.sda3.out‚ 34.45703125 disk.rbd0.out‚0 disk.dm-0.in‚ 1949.234375" disk.dm-0.out‚3879.75244140625( proc.memory.availableMB‚ 198.32421875 network.sent‚ 4.154522e+06 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 disk.sda.in‚ 0.10546875 disk.nvme0n1.in‚ 1949.234375% disk.nvme0n1.out‚3879.75244140625 disk.rbd0.in‚0 disk.sda2.out‚0) disk./.usagePercent‚39.61044802997902$ disk./.usageGB‚590.2243003845215 disk.sda2.in‚0‚£;œ— ‚(:‘ ãíÒ̘ÁŠ gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚181.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.233599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜³e4˘ ‚(:À ãíÒ¢îŒ disk.nvme0n1.in‚ 1949.234375 disk.sda2.out‚0 disk.rbd1.out‚3.75! disk.nvme0c0n1.in‚ 1949.234375 disk.sda3.in‚ 0.10546875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚26.92436964196056* disk./.usagePercent‚39.610427549376276 disk.sda2.in‚0 disk.dm-1.in‚ 0.10546875$ memory_percent‚98.79059791564941 disk.rbd1.in‚0% disk.nvme0n1.out‚3947.80126953125 disk.dm-1.out‚34.6875 disk.dm-0.in‚ 1949.234375 disk.sda.out‚34.6875 network.recv‚523509$ disk./.usageGB‚590.2239952087402' disk.nvme0c0n1.out‚3947.80126953125# proc.memory.rssMB‚ 2691.11328125 disk.sda1.out‚0 disk.sda3.out‚34.6875 disk.sda.in‚ 0.10546875' proc.memory.availableMB‚ 198.1484375* proc.memory.percent‚16.425251960754395 network.sent‚ 4.211745e+06 disk.rbd0.in‚0" disk.dm-0.out‚3947.80126953125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 À¿2™ - _wandbruntime‚1548  _runtime‚1548G€ †Çš Á & lr-AdamW/pg2‚0.0008399465806486894 trainer/global_step‚3249 % _timestamp‚1.7843696067935622e+09 _runtime‚1548.237368446 _step‚260 & lr-AdamW/pg1‚0.0008399465806486894…T·ºÌ› Æ _step‚260 & lr-AdamW/pg1‚0.0008399465806486894 & lr-AdamW/pg2‚0.0008399465806486894 trainer/global_step‚3249 % _timestamp‚1.7843696067935622e+09 _runtime‚1548.237368446„ k`2œ - _wandbruntime‚1548  _runtime‚15487*lTÕ ÇO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1918853447168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126382023441920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7393610 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14992768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1912105186084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 $ _timestamp‚1.784369606794706e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119118804998656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55264853 _runtime‚1548.238925437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1918765510144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30864521 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14992765 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117135983562752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1982821435904 ?9DeviceSt<ê|…øatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122372113119232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚126382128418304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚3249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117136008466432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124463274971136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55274832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124463257931776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1918765510144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122372096079872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124463274971136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24410311 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126382128418304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24410309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1983001934336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126382023441920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30864521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30854544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7393843 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124463257931776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1918853447168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30854544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128’ìkÒ'ž ÌO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117136008466432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124463274971136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55274832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124463257931776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1918765510144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122372096079872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124463274971136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24410311 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126382128418304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24410309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1983001934336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126382023441920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30864521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30854544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7393843 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124463257931776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1918853447168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30854544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1918853447168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126382023441920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7393610 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14992768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1912105186084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 $ _timestamp‚1.784369606794706e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119118804998656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55264853 _runtime‚1548.238925437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1918765510144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30864521 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14992765 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117135983562752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1982821435904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122372113119232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚126382128418304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚3249…xçò*2Ÿ - _wandbruntime‚1548  _runtime‚1548àÿ­×&  ÑM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1919422813696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24417532 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24417532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1983408087552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 $ _timestamp‚1.784369607132708e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1919334869504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55291348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30863825 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1919422813696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22393042 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30873816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124321257662286 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1912586370524 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119154506462720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55291348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14997252 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122408568115200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126419808749056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117170895077376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24417530 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7396025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124500385935360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55281355 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124500385935360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126419703765504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14997249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117170919981056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚3249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7395793 _runtime‚1548.576796991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124321154485724 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1919334869504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126419808749056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24417530 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22393277 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1983586481664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122408585154560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30863825 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119154303164928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124500368896000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¡×7 ¡ ÖM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117170895077376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24417530 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7396025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124500385935360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55281355 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124500385935360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126419703765504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14997249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117170919981056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚3249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7395793 _runtime‚1548.576796991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124321154485724 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1919334869504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126419808749056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24417530 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22393277 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1983586481664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122408585154560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30863825 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119154303164928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124500368896000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1919422813696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24417532 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24417532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1983408087552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 $ _timestamp‚1.784369607132708e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1919334869504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55291348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30863825 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1919422813696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22393042 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30873816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124321257662286 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1912586370524 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119154506462720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55291348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14997252 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122408568115200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126419808749056†DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15216968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126318823182336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1947353672192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128266071878144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2012239522816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1940503165792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚3299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126318823182336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24771359 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31320446 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120893798816768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126318806142976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24771361 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22720702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126136218051424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128266176854528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124195697846272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24771361 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128266071878144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7503737 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24771359 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56101784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947353672192ã êWÓ'¬ ÍO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24771361 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128266071878144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7503737 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24771359 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56101784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947353672192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947265735168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31320446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15216965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7503978 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120894004218880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31330423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22720946 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118881559293952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118881584197632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2012420021248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56091805 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1947265735168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126318806142976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15216968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126318823182336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1947353672192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128266071878144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2012239522816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1940503165792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚3299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126318823182336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24771359 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31320446 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120893798816768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126318806142976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24771361 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22720702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126136218051424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128266176854528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124195697846272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146‰©YÑL2­ - _wandbruntime‚1571  _runtime‚1571dyÑ|Z® ÒM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31339718 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7505928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24778580 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31329727 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120929503049728 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚124232169881600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1941070487434 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126355934146560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22727377 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31329727 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118916470808576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126173154231832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1940984350232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56118300 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24778582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126173257408394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚266 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1947835094528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56108307 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7506158 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2013007337472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24778582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128303752201728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120929291363328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.onx»¹p~ _train_batch_end/allocated_bytes.all.allocated‚128303857185280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1947835094528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126355934146560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128303857185280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126355917107200 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1947923038720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2012820554752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15221452 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186782720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843696297810874e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1947923038720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2012820554752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15221452 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186782720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843696297810874e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24778580 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31329727 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120929503049728 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚124232169881600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1941070487434 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126355934146560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22727377 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31329727 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118916470808576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126173154231832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1940984350232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56118300 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24778582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126173257408394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚266 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1947835094528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56108307 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7506158 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2013007337472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24778582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128303752201728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120929291363328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128303857185280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1947835094528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936ŠVq"ª2° - _wandbruntime‚1571  _runtime‚1571&5ü±  2loss/FBt2/assignment_loss‚0.029109452205579166 trainer/global_step‚3299 2loss/SRqqt2/assignment_loss‚0.6617673961792893 3loss/FBt1/assignment_loss‚0.0012707229239596884 -loss/FRt1/detection_loss‚0.56209478003372 1loss/SRqqt1/detection_loss‚1.0516549329701308 /loss/FBt2/detection_loss‚0.6600885998108424 %loss/total_loss‚9.766554554542925 epoch‚4 _runtime‚1571.226252473 /loss/FRt1/assignment_loss‚2.004304613368179 0loss/FRt2/assignment_loss‚1.1845681402148032 1loss/SRqqt2/detection_loss‚0.7160875506281079 /loss/FBt1/detection_loss‚0.8369976792018861 % _timestamp‚1.7843696297827141e+09 _step‚267 2loss/SRqqt1/assignment_loss‚0.6519599480637447 /loss/FRt2/detection_loss‚1.4066507389426839Üâž ² † 2loss/SRqqt1/assignment_loss‚0.6519599480637447 /loss/FRt2/detection_loss‚1.4066507389426839 2loss/FBt2/assignment_loss‚0.029109452205579166 trainer/global_step‚3299 2loss/SRqqt2/assignment_loss‚0.6617673961792893 3loss/FBt1/assignment_loss‚0.0012707229239596884 -loss/FRt1/detection_loss‚0.56209478003372 1loss/SRqqt1/detection_loss‚1.0516549329701308 /loss/FBt2/detection_loss‚0.6600885998108424 %loss/total_loss‚9.766554554542925 epoch‚4 _runtime‚1571.226252473 /loss/FRt1/assignment_loss‚2.004304613368179 0loss/FRt2/assignment_loss‚1.1845681402148032 1loss/SRqqt2/detection_loss‚0.7160875506281079 /loss/FBt1/detection_loss‚0.8369976792018861 % _timestamp‚1.7843696297827141e+09 _step‚267‹^&›#œ³ ‚(:‘ á£íÒ˜Ò㊠gpu.0.gpu‚64.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.111600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á0mtß´ ‚(:Ô á£íÒ†’±Œ( proc.memory.availableMB‚ 198.33203125$ disk./.usageGB‚590.3144798278809 disk.dm-1.in‚ 0.10546875" disk.dm-0.out‚3975.47802734375 disk.sda3.in‚ 0.10546875# proc.memory.rssMB‚ 2691.11328125 network.recv‚532010* proc.memory.percent‚16.425251960754395 cpu‚26.93438159363311 proc.cpu.threads‚17' disk.nvme0c0n1.out‚3975.47802734375 disk.sda.in‚ 0.10546875 disk.sda.out‚ 37.01953125 disk.sda1.out‚0 disk.sda3.out‚ 37.01953125 network.sent‚ 4.279412e+06# disk.nvme0c0n1.in‚ 2110.66796875% disk.nvme0n1.out‚3975.47802734375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚3.875$ memory_percent‚98.78947734832764 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 37.01953125 disk.rbd1.in‚0* disk./.usagePercent‚39.616500048088454! disk.nvme0n1.in‚ 2110.66796875 disk.dm-0.in‚ 2110.66796875 disk.drbd2165.out‚0!ѵ ‚(:Æ ð£íÒ‚àŒ proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0 cpu‚26.938686331759545 disk.rbd0.out‚0 disk.sda.in‚ 0.10546875 network.sent‚ 4.342311e+06* proc.memory.percent‚16.425251960754395$ disk./.usageGB‚590.2841453552246 disk.sda.out‚37.71875 disk.rbd0.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 198.12890625 disk.rbd1.out‚3.9375# disk.nvme0c0n1.in‚ 2110.66796875# disk.nvme0n1.out‚4029.912109375 disk.sda2.out‚0 disk.dm-1.out‚37.71875 disk.dm-0.out‚4029.912109375 disk.sda3.out‚37.71875 disk.drbd2165.out‚0 disk.dm-0.in‚ 2110.66796875% disk.nvme0c0n1.out‚4029.912109375! disk.nvme0n1.in‚ 2110.66796875 disk.rbd1.in‚0 network.recv‚537114) disk./.usagePercent‚39.61446427617618 disk.sda1.out‚0$ memory_percent‚98.79071712493896# proc.memory.rssMB‚ 2691.11328125+Ùþ‘¶ ‚(:† ð£íÒÉÊôŸ gpu.0.gpu‚60.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚164.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03†ÄÙ2· - _wandbruntime‚1593  _runtime‚1593èò DǸ Á trainer/global_step‚3349 % _timestamp‚1.7843696521397207e+09 _runtime‚1593.583596226 _step‚268 & lr-AdamW/pg1‚0.0008392724669277206 & lr-AdamW/pg2‚0.00083927246692772068m2ŒÌ¹ Æ & lr-AdamW/pg2‚0.0008392724669277206 trainer/global_step‚3349 % _timestamp‚1.7843696521397207e+09 _runtime‚1593.583596226 _step‚268 & lr-AdamW/pg1‚0.0008392724669277206Œ±rˆ§2º - _wandbruntime‚1593  _runtime‚1593šÀ€}Î'» ÈO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130150120314368 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25132409 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126019316652032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56928736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127988217797532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31786348 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128174354354176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2041623016448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120627135025152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127988114621082 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128174354354176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15441168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975853897216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2041795126272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1968901145500 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25132409 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7613667 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23055054 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15441165 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31796325 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1975765960192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975853897216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1968815008410 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25132411 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7613886 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÌTáÅU¼ ÍO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23055054 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15441165 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31796325 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1975765960192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975853897216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1968815008410 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25132411 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7613886 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130150120314368 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25132409 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126019316652032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56928736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127988217797532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31786348 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128174354354176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2041623016448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120627135025152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127988114621082 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128174354354176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15441168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975853897216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2041795126272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1968901145500 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25132409 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7613667 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936J³â½2½ - _wandbruntime‚1593  _runtime‚1593˜¼Vh×&¾ ÑM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1976335319552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56945252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161616896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128211482357760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186520576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7615855 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31795629 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2042379812864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128025257154502 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130187800637952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1969468467142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120662046539776 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25139632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2042218195968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15445652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7616067 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 $ _timestamp‚1.784369652483312e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56945252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56935259 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23061504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1976423263744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31805620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25139630 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128025153977940 trainer/global_step‚3349 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56935259 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126055771648000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128211465318400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0&êÏÜ&¿ ÖM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚130187905621504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1976423263744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122704451256320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1969382329940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128211465318400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31795629 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120662071443456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130187905621504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120662046539776 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25139632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2042218195968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15445652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7616067 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 $ _timestamp‚1.784369652483312e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56945252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56935259 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23061504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1976423263744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31805620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25139630 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128025153977940 trainer/global_step‚3349 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56935259 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126055771648000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128211465318400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1976335319552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56945252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161616896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128211482357760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186520576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7615855 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31795629 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2042379812864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128025257154502 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130187800637952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1969468467142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FB ˆ 0loss/FRt1/assignment_loss‚1.9150545879479974 3loss/FBt1/assignment_loss‚0.0030664331943360897 /loss/FRt2/detection_loss‚1.3238776305919764 1loss/SRqqt1/detection_loss‚0.9529685793783177 /loss/FBt1/detection_loss‚0.6046203648050626 2loss/SRqqt1/assignment_loss‚0.6902107598786181 %loss/total_loss‚9.358379181464112 _step‚271 0loss/SRqqt2/detection_loss‚0.711630323948202 trainer/global_step‚3349 0loss/FRt2/assignment_loss‚1.1200051917068994 2loss/SRqqt2/assignment_loss‚0.5953740917405324 3loss/FBt2/assignment_loss‚0.0036263507322575423 /loss/FRt1/detection_loss‚0.6090543866680539 /loss/FBt2/detection_loss‚0.8288904808718582 epoch‚4 $ _timestamp‚1.784369652485461e+09 _runtime‚1593.9289922383î3Là ‚(:„ ÿ£íÒ·ÄŒ gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚177.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 YTÚÝÄ ‚(:Ò ÿ£íÒÅûÖŽ disk.sda3.out‚ 38.49609375 disk.rbd0.in‚0 disk.dm-1.in‚ 0.10546875% disk.nvme0n1.out‚4040.63720703125 disk.sda2.in‚0# proc.memory.rssMB‚ 2691.11328125 proc.cpu.threads‚17$ disk./.usageGB‚590.2841949462891 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 198.34375* disk./.usagePercent‚39.614467604274125 disk.rbd1.out‚3.9375' disk.nvme0c0n1.out‚4040.63720703125! disk.nvme0n1.in‚ 2110.66796875 disk.dm-1.out‚ 38.49609375* proc.memory.percent‚16.425251960754395 disk.sda.out‚ 38.49609375 disk.drbd2165.in‚0 disk.dm-0.in‚ 2110.66796875# memory_percent‚98.7894058227539 disk.sda.in‚ 0.10546875 disk.sda1.out‚0" disk.dm-0.out‚4040.63720703125# disk.nvme0c0n1.in‚ 2110.66796875 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚26.943317761628947 network.sent‚ 4.360248e+06 network.recv‚540950 disk.sda3.in‚ 0.10546875Rs-ì2Å - _wandbruntime‚1616  _runtime‚1616l¬ÏTÇÆ Á & lr-AdamW/pg1‚0.0008389254309234659 & lr-AdamW/pg2‚0.0008389254309234659 trainer/global_step‚3399 % _timestamp‚1.7843696748376672e+09 _runtime‚1616.281574655 _step‚272B2ÌÇ Æ % _timestamp‚1.7843696748376672e+09 _runtime‚1616.281574655 _step‚272 & lr-AdamW/pg1‚0.0008389254309234659 & lr-AdamW/pg2‚0.0008389254309234659 trainer/global_step‚3399Їa’2È - _wandbruntime‚1616  _runtime‚1616Œ÷3 É ÈO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122372710756352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3399 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1997212988118 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2004354122240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32252250 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _runtime‚1616.282856364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132034168750592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2004266185216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 % _timestamp‚1.7843696748388565e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124444022708224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124443834083328 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25493461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15665368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23388415 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23388196 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitŽj´?or.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25493459 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7722831 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15665365 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32252250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2071287048192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129840114367190 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130029919604736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122372735660032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129840217543640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127842918418432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130029919604736 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚132034273726976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2004266185216 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132034273726976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132034168750592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32262227 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25493459 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32262227 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128ø¢$Ó'Ê ÍO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130029919604736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122372735660032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129840217543640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127842918418432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130029919604736 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚132034273726976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2004266185216 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132034273726976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132034168750592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32262227 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25493459 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32262227 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122372710756352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3399 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1997212988118 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2004354122240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32252250 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _runtime‚1616.282856364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132034168750592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2004266185216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 % _timestamp‚1.7843696748388565e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124444022708224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124443834083328 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25493461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15665368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23388415 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23388196 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25493459 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7722831 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15665365 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32252250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2071287048192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129840114367190‘¤ðøØ2Ë - _wandbruntime‚1616  _runtime‚1616I9^bØ&Ì ÒM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25500680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32271522 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7725257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2004923488768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124479528256512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25500682 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32271522 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7725018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127879390453760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130067013529600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32261531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15669849 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57762211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2071881081856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15669852 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1997866446850 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32261531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2004923488768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57772204 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25500680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124479329153024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2004835544576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25500682 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127879373414400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2004835544576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57772204 _step‚274 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130067030568960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132071954057728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23394867 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1616.625201501 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129877153724048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130067030568960 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122407647174656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23395109 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130067013529600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2071706882048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1997780309648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327UEÁÍ ×M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57762211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2071881081856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15669852 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1997866446850 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32261531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2004923488768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57772204 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25500680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124479329153024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2004835544576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25500682 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127879373414400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2004835544576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57772204 _step‚274 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130067030568960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132071954057728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23394867 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1616.625201501 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129877153724048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130067030568960 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122407647174656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23395109 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes:k®Þ.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130067013529600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2071706882048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1997780309648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25500680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32271522 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7725257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2004923488768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124479528256512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25500682 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32271522 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7725018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127879390453760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130067013529600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32261531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15669849 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640’8iS2Î - _wandbruntime‚1616  _runtime‚1616È;êÏ  0loss/FRt1/assignment_loss‚2.0388451500396063 2loss/SRqqt1/assignment_loss‚0.5720341923774895 2loss/SRqqt2/assignment_loss‚0.6372789560637236 /loss/FRt2/detection_loss‚1.2126356253194648 1loss/SRqqt1/detection_loss‚0.8833175806473651 1loss/SRqqt2/detection_loss‚0.7844511925278556 trainer/global_step‚3399 $ _timestamp‚1.784369675183243e+09 /loss/FRt1/detection_loss‚0.6489380585767114 /loss/FBt1/detection_loss‚0.9681241240973274 epoch‚4 _runtime‚1616.626783006 _step‚275 0loss/FRt2/assignment_loss‚1.0330175944928373 %loss/total_loss‚9.372552009493248 1loss/FBt1/assignment_loss‚0.06213651752516777 2loss/FBt2/assignment_loss‚0.026217582941461457 /loss/FBt2/detection_loss‚0.5055554348842374q25 Ð † 2loss/SRqqt1/assignment_loss‚0.5720341923774895 2loss/SRqqt2/assignment_loss‚0.6372789560637236 /loss/FRt2/detection_loss‚1.2126356253194648 1loss/SRqqt1/detection_loss‚0.8833175806473651 1loss/SRqqt2/detection_loss‚0.7844511925278556 trainer/global_step‚3399 $ _timestamp‚1.784369675183243e+09 /loss/FRt1/detection_loss‚0.6489380585767114 /loss/FBt1/detection_loss‚0.9681241240973274 epoch‚4 _runtime‚1616.626783006 _step‚275 0loss/FRt2/assignment_loss‚1.0330175944928373 %loss/total_loss‚9.372552009493248 1loss/FBt1/assignment_loss‚0.06213651752516777 2loss/FBt2/assignment_loss‚0.026217582941461457 /loss/FBt2/detection_loss‚0.5055554348842374 0loss/FRt1/assignment_loss‚2.0388451500396063“Ó*¥‘Ñ ‚(:† Ž¤íÒŒíÀŠ gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÂ·)ÔÒ ‚(:É Ž¤íÒ©¶´ disk.sda.out‚ 38.70703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚546010 disk.rbd1.out‚4 disk.dm-1.out‚ 38.70703125 disk.dm-0.in‚ 2211.1328125 disk.sda3.out‚ 38.70703125$ disk.nvme0n1.out‚4045.0849609375 disk.rbd0.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 198.2890625 cpu‚26.947298976546147 network.sent‚ 4.410092e+06$ disk./.usageGB‚590.2843017578125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 disk.nvme0n1.in‚ 2211.1328125 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2691.11328125) disk./.usagePercent‚39.61447477248508 disk.sda.in‚ 0.10546875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 2211.1328125$ memory_percent‚98.78973960876465* proc.memory.percent‚16.425251960754395! disk.dm-0.out‚4045.0849609375& disk.nvme0c0n1.out‚4045.0849609375Bl‘Ó ‚(:† ¤íÒı¿Š gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÜ¤–ÙÔ ‚(:Î ¤íÒ°Ž§Œ disk.sda.out‚ 38.87109375" disk.nvme0c0n1.in‚ 2211.1328125 disk.sda1.in‚0( proc.memory.availableMB‚ 133.37109375 disk.dm-0.in‚ 2211.1328125$ disk./.usageGB‚590.2835502624512 disk.rbd1.out‚4% disk.nvme0n1.out‚4051.74560546875 disk.sda1.out‚0 cpu‚26.952088927163636 disk.drbd2165.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2691.11328125 disk.rbd0.in‚0 disk.sda2.in‚0" disk.dm-0.out‚4051.74560546875$ memory_percent‚99.18596744537354 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875* disk./.usagePercent‚39.614424339000834 disk.sda.in‚ 0.10546875' disk.nvme0c0n1.out‚4051.74560546875 disk.nvme0n1.in‚ 2211.1328125 disk.sda3.in‚ 0.10546875 disk.sda3.out‚ 38.87109375 proc.cpu.threads‚17 network.sent‚ 4.472656e+06 network.recv‚551187 disk.dm-1.out‚ 38.87109375* proc.memory.percent‚16.425251960754395`¡@N2Õ - _wandbruntime‚1639  _runtime‚1639whc±ÅÖ ¿ % lr-AdamW/pg1‚0.000838571749532503 % lr-AdamW/pg2‚0.000838571749532503 trainer/global_step‚3449 % _timestamp‚1.7843696975691488e+09 _runtime‚1639.012968795 _step‚276[äÅ“Ê× Ä % lr-AdamW/pg1‚0.000838571749532503 % lr-AdamW/pg2‚0.000838571749532503 trainer/global_step‚3449 % _timestamp‚1.7843696975691488e+09 _runtime‚1639.012968795 _step‚276”âU 02Ø - _wandbruntime‚1639  _runtime‚1639ïN`uÌ'Ù ÆO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2025697104916 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32728129 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32718152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15889565 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133918322163200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2032854347264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7831508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133918217186816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25854509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23721073 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2100830086144Ñ#«ÈÑ'Ú ËO _runtime‚ 1639.01420973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25854509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25854511 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2032854347264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23721308 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129666520184832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25854511 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2032766410240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25854509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23721073 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2100830086144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2025697104916 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32728129 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32718152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15889565 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133918322163200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2032854347264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7831508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133918217186816 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2033423713792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2033335769600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23727746 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7833929 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101246585856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131729153470156 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32737424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131922561740800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126254444588032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133956002493952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7833697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_?5 B8bytes.large_pool.freed‚131922561740800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _step‚278 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131922578780160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124153198002176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25861730 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129702975180800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131922578780160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58589163 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129702992220160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _runtime‚1639.359979973 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58599156 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25861732 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101429174272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15894049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25861732 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58589163 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32727433 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32727433 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15894052 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552âsDDÛ&Ý ÕM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131922561740800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _step‚278 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131922578780160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124153198002176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25861730 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129702975180800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131922578780160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58589163 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129702992220160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _runtime‚1639.359979973 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58599156 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25861732 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101429174272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15894049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25861732 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58589163 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32727433 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32727433 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15894052 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2026178289356 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2033423713792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2033335769600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23727746 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7833929 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101246585856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131729153470156 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32737424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131922561740800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126254444588032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133956002493952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7833697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0– »>¥2Þ - _wandbruntime‚1639  _runtime‚1639àôÙß € 1loss/FBt1/assignment_loss‚0.07484094994086474 0loss/SRqqt1/detection_loss‚1.056697977671198 epoch‚4 _step‚279 2loss/SRqqt1/assignment_loss‚0.6705449163673852 2loss/FBt2/assignment_loss‚0.015745453011274644 /loss/FRt1/detection_loss‚0.6291359531720453 /loss/FRt2/detection_loss‚1.4151461060887935 1loss/SRqqt2/detection_loss‚0.6982771810298549 /loss/FBt2/detection_loss‚0.5979938468447438 _runtime‚1639.361198135 0loss/FRt1/assignment_loss‚1.7622508634313494 /loss/FRt2/assignment_loss‚1.037009489096579 % _timestamp‚1.7843696979176676e+09 /loss/FBt1/detection_loss‚1.2086807133799249 %loss/total_loss‚9.738351360056996 trainer/global_step‚3449 2loss/SRqqt2/assignment_loss‚0.5720279100229837:yÓ à … _runtime‚1639.361198135 0loss/FRt1/assignment_loss‚1.7622508634313494 /loss/FRt2/assignment_loss‚1.037009489096579 % _timestamp‚1.7843696979176676e+09 /loss/FBt1/detection_loss‚1.2086807133799249 %loss/total_loss‚9.738351360056996 trainer/global_step‚3449 2loss/SRqqt2/assignment_loss‚0.5720279100229837 1loss/FBt1/assignment_loss‚0.07484094994086474 0loss/SRqqt1/detection_loss‚1.056697977671198 epoch‚4 _step‚279 2loss/SRqqt1/assignment_loss‚0.6705449163673852 2loss/FBt2/assignment_loss‚0.015745453011274644 /loss/FRt1/detection_loss‚0.6291359531720453 /loss/FRt2/detection_loss‚1.4151461060887935 1loss/SRqqt2/detection_loss‚0.6982771810298549 /loss/FBt2/detection_loss‚0.5979938468447438—¾Üé‘á ‚(:† ¬¤íÒøØ‹ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚183.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x*‡ªÔâ ‚(:É ¬¤íÒ™²…" disk.nvme0c0n1.in‚ 2211.1328125& disk.nvme0c0n1.out‚4054.9775390625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.out‚ 39.38671875 disk.drbd2165.in‚0% proc.memory.availableMB‚ 133.46875 proc.cpu.threads‚17 disk.rbd1.in‚0$ disk.nvme0n1.out‚4054.9775390625 disk.sda1.out‚0 network.recv‚554653) disk./.usagePercent‚39.61448194069604 disk.rbd1.out‚4.0625 disk.nvme0n1.in‚ 2211.1328125 disk.drbd2165.out‚0* proc.memory.percent‚16.425251960754395 disk.dm-1.out‚ 39.38671875 cpu‚26.95562178740858 disk.sda.out‚ 39.38671875 disk.dm-0.in‚ 2211.1328125# proc.memory.rssMB‚ 2691.11328125 network.sent‚ 4.4776e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.10546875$ memory_percent‚99.18537139892578$ disk./.usageGB‚590.2844085693359 disk.sda1.in‚0 disk.sda2.in‚0! disk.dm-0.out‚4054.9775390625 disk.sda.in‚ 0.10546875®—Ê2ã - _wandbruntime‚1661  _runtime‚1661¯“ªÆä À & lr-AdamW/pg2‚0.0008382114284197572 trainer/global_step‚3499 $ _timestamp‚1.784369719852673e+09 _runtime‚1661.296605406 _step‚280 & lr-AdamW/pg1‚0.0008382114284197572H~VÒËå Å & lr-AdamW/pg1‚0.0008382114284197572 & lr-AdamW/pg2‚0.0008382114284197572 trainer/global_step‚3499 $ _timestamp‚1.784369719852673e+09 _runtime‚1661.296605406 _step‚280˜2™<2æ - _wandbruntime‚1661  _runtime‚1661}ÚýÎ'ç ÈO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26215559 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131490121951232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33194031 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135802265623040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26215561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16113765 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7940432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33194031 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133741016027136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061266635264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 trainer/global_step‚3499 _step‚281 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135802370599424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26215561 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7940659 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127994196420096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2061354572288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135802370599424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16113768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133740998987776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231ç]A'è ÍO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125863887122432 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59399613 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133740998987776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131490104911872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33184054 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133741016027136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2054095084624 _runtime‚1661.297857432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2061354572288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24054427 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127993999406592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ìDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135802370599424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26215561 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7940659 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127994196420096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2061354572288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135802370599424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16113768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133740998987776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26215559 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131490121951232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33194031 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135802265623040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26215561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16113765 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7940432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33194031 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133741016027136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061266635264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 trainer/global_step‚3499 _step‚281 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133778109952000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2130738398208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131526576947200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2054662406266 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133581256392826 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2061835994624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26222782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 180229120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131526593986560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚3499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125898773733376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 1661.64271854 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59426108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33193335 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33203326 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135840050930176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125898798637056 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133778109952000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128029692360704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16118252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33203326 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59426108 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128029512131584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59416115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24060852 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155325440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133778126991360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24061075 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26222782 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59416115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2061923938816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2061835994624„*&Ü&ë ÖM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135840050930176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125898798637056 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133778109952000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128029692360704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16118252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33203326 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59426108 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128029512131584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59416115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24060852 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155325440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133778126991360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24061075 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26222782 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59416115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2061923938816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2061835994624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2130893723648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133778109952000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2130738398208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131526576947200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2054662406266 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133581256392826 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2061835994624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26222782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 180229120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131526593986560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚3499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125898773733376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _runtime‚ 1661.64271854 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59426108 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33193335 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33203326šK¬q2ì - _wandbruntime‚1661  _runtime‚1661èÂÙí ‚ 0loss/FRt2/assignment_loss‚1.1392400413751602 2loss/FBt1/assignment_loss‚0.029640910565583246 1loss/FBt2/assignment_loss‚0.01384990787202804 0loss/FBt2/detection_loss‚0.42146157901323233 %loss/total_loss‚9.058609498389554 epoch‚4 % _timestamp‚1.7843697202002149e+09 _runtime‚1661.644094492 2loss/SRqqt2/assignment_loss‚0.6294836470419746 .loss/FRt1/detection_loss‚0.694058963414371 1loss/SRqqt1/detection_loss‚0.9509233195060418 1loss/SRqqt2/detection_loss‚0.6581766777518359 _step‚283 /loss/FRt2/detection_loss‚1.3212807054187579 /loss/FBt1/detection_loss‚0.4768938916650685 trainer/global_step‚3499 0loss/FRt1/assignment_loss‚1.9652765626896729 2loss/SRqqt1/assignment_loss‚0.7583232920758277°1ƒ î ‡ /loss/FRt2/detection_loss‚1.3212807054187579 /loss/FBt1/detection_loss‚0.4768938916650685 trainer/global_step‚3499 0loss/FRt1/assignment_loss‚1.9652765626896729 2loss/SRqqt1/assignment_loss‚0.7583232920758277 0loss/FRt2/assignment_loss‚1.1392400413751602 2loss/FBt1/assignment_loss‚0.029640910565583246 1loss/FBt2/assignment_loss‚0.01384990787202804 0loss/FBt2/detection_loss‚0.42146157901323233 %loss/total_loss‚9.058609498389554 epoch‚4 % _timestamp‚1.7843697202002149e+09 _runtime‚1661.644094492 2loss/SRqqt2/assignment_loss‚0.6294836470419746 .loss/FRt1/detection_loss‚0.694058963414371 1loss/SRqqt1/detection_loss‚0.9509233195060418 1loss/SRqqt2/detection_loss‚0.6581766777518359 _step‚283›WÁ|ê‘ï ‚(:† »¤íÒû²ä‹ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚135.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0).µýÚð ‚(:Ï »¤íÒ²ºà# proc.memory.rssMB‚ 2691.11328125 cpu‚26.959683970911858 network.sent‚ 4.540246e+06 disk.sda.in‚ 0.10546875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 2211.1328125" disk.nvme0c0n1.in‚ 2211.1328125$ disk.nvme0n1.out‚4060.3642578125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚4.0625 network.recv‚559763! disk.dm-0.out‚4060.3642578125 disk.sda.out‚ 39.62890625 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚4060.3642578125 disk.rbd0.out‚0$ disk./.usageGB‚590.2844619750977 disk.sda3.in‚ 0.10546875 disk.dm-1.in‚ 0.10546875$ memory_percent‚99.18694496154785 disk.rbd1.in‚0* disk./.usagePercent‚39.614485524801516 disk.dm-0.in‚ 2211.1328125* proc.memory.percent‚16.425251960754395 disk.sda3.out‚ 39.62890625 disk.dm-1.out‚ 39.62890625' proc.memory.availableMB‚ 133.2109375ŠÀˆ_›ñ ‚(: ʤíҕʪŠ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚175.909' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.181799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„£ãìÔò ‚(:É ʤíÒ­¤õ" disk.dm-0.out‚4063.48486328125 disk.sda.in‚ 0.10546875 disk.nvme0n1.in‚ 2305.265625# proc.memory.rssMB‚ 2691.11328125 disk.dm-1.in‚ 0.10546875' disk.nvme0c0n1.out‚4063.48486328125 disk.rbd0.out‚0 disk.dm-0.in‚ 2305.265625 disk.sda3.in‚ 0.10546875% proc.memory.availableMB‚ 133.65625 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚26.96354661194903 disk.rbd0.in‚0 disk.sda2.out‚0# memory_percent‚99.1842269897461 disk.sda3.out‚ 39.90234375 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 2305.265625% disk.nvme0n1.out‚4063.48486328125 proc.cpu.threads‚17 network.recv‚565005 disk.dm-1.out‚ 39.90234375 disk.drbd2165.in‚0* proc.memory.percent‚16.425251960754395 network.sent‚ 4.602979e+06$ disk./.usageGB‚590.2844276428223 disk.sda1.out‚0 disk.sda.out‚ 39.90234375 disk.drbd2165.out‚0 disk.rbd1.out‚4.125) disk./.usagePercent‚39.61448322073371Mê¤;2ó - _wandbruntime‚1683  _runtime‚1683D^|-Áô Á % _timestamp‚1.7843697423966243e+09 _runtime‚1683.840514247 _step‚284 & lr-AdamW/pg1‚0.0008378444733565027 & lr-AdamW/pg2‚0.0008378444733565027 trainer/global_step‚Ã…îs3549¶O%Ìõ Æ & lr-AdamW/pg1‚0.0008378444733565027 & lr-AdamW/pg2‚0.0008378444733565027 trainer/global_step‚3549 % _timestamp‚1.7843697423966243e+09 _runtime‚1683.840514247 _step‚284œÑsä2ö - _wandbruntime‚1683  _runtime‚1683"4ùïÎ'÷ ÈO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2089854797312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129769247843840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127609437949952 trainer/global_step‚3549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137686314059264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135596547198976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2089854797312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133313706678272 _step‚285 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135396216781964 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135596564238336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2159784990208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60226565 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135596547198976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33659933 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24387331 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2159602394624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33649956 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129769040344576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2082406927242 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8049134 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2082493064332 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2089766860288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135396113605514 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146%ëäÓ'ø ÍO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137686314059264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135596547198976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2089854797312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133313706678272 _step‚285 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135396216781964 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135596564238336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2159784990208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60226565 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135596547198976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33659933 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24387331 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2159602394624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33649956 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129769040344576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2082406927242 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8049134 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2082493064332 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2089766860288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135396113605514 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2089854797312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129769247843840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127609437949952 trainer/global_step‚3549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488ì¢ 2ù - _wandbruntime‚1684  _runtime‚1684åÀGÂØ&ú ÒM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33669228 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135633675202560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26583832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135433256138934 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133350195752960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16342452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129804546252288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16342449 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2160362598912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33659237 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137724099366400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33669228 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60253060 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137724099366400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135633658163200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135633658163200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26583830 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2090424163840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133350178713600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 % _timestamp‚1.7843697427389097e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60253060 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2090424163840 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2090336219648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129804736967168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24394013 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2082974248772 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26583832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26583830 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24393778 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127644374368256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473Çvû ×M C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2090336219648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129804736967168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24394013 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2082974248772 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26583832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26583830 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24393778 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127644374368256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33669228 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135633675202560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.all˜—›¶Äocated‚26583832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135433256138934 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133350195752960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16342452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129804546252288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16342449 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2160362598912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33659237 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137724099366400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33669228 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60253060 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137724099366400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135633658163200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135633658163200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26583830 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2090424163840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133350178713600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 % _timestamp‚1.7843697427389097e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60253060 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2090424163840ž˜u ÿ2ü - _wandbruntime‚1684  _runtime‚1684æMä ý ƒ 2loss/SRqqt2/assignment_loss‚0.5412193878930928 .loss/FRt2/detection_loss‚1.185499223547023 1loss/SRqqt1/detection_loss‚1.0262467604713374 1loss/SRqqt2/detection_loss‚0.6886191075589035 _step‚287 0loss/FRt1/assignment_loss‚1.8909299487520934 2loss/SRqqt1/assignment_loss‚0.6005191235952745 /loss/FBt1/detection_loss‚0.6294371166399547 #loss/total_loss‚8.8069931905391 epoch‚4 0loss/FRt2/assignment_loss‚1.1550571729029928 4loss/FBt1/assignment_loss‚0.00030816195952963554 2loss/FBt2/assignment_loss‚0.002759346354507327 /loss/FRt1/detection_loss‚0.6530348195129937 0loss/FBt2/detection_loss‚0.43336302135139704 _runtime‚1684.184376914 trainer/global_step‚3549 % _timestamp‚1.7843697427408433e+09T±\þ ˆ 0loss/FRt1/assignment_loss‚1.8909299487520934 2loss/SRqqt1/assignment_loss‚0.6005191235952745 /loss/FBt1/detection_loss‚0.6294371166399547 #loss/total_loss‚8.8069931905391 epoch‚4 0loss/FRt2/assignment_loss‚1.1550571729029928 4loss/FBt1/assignment_loss‚0.00030816195952963554 2loss/FBt2/assignment_loss‚0.002759346354507327 /loss/FRt1/detection_loss‚0.6530348195129937 0loss/FBt2/detection_loss‚0.43336302135139704 _runtime‚1684.184376914 trainer/global_step‚3549 % _timestamp‚1.7843697427408433e+09 2loss/SRqqt2/assignment_loss‚0.5412193878930928 .loss/FRt2/detection_loss‚1.185499223547023 1loss/SRqqt1/detection_loss‚1.0262467604713374 1loss/SRqqt2/detection_loss‚0.6886191075589035 _step‚287ŸÚ ß‹‘ÿ ‚(:† Ù¤íÒμ¹Š gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó‘öÍ€ ‚(: ٤íÒœ disk.nvme0n1.in‚ 2305.265625 network.sent‚ 4.607824e+06 disk.sda2.out‚0 disk.dm-0.in‚ 2305.265625 disk.sda3.in‚ 0.10546875$ disk.nvme0n1.out‚4070.4384765625 disk.sda3.out‚40.09375& disk.nvme0c0n1.out‚4070.4384765625$ memory_percent‚99.18568134307861* disk./.usagePercent‚39.614461460093295 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚40.09375 disk.rbd1.out‚4.125$ disk./.usageGB‚590.2841033935547 disk.rbd0.out‚0# proc.memory.rssMB‚ 2691.11328125 disk.sda1.out‚0 disk.sda.in‚ 0.10546875 disk.rbd1.in‚0* proc.memory.percent‚16.425251960754395 disk.sda2.in‚0! disk.dm-0.out‚4070.4384765625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 133.41796875 cpu‚26.96809818086267 proc.cpu.threads‚17 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 2305.265625 disk.rbd0.in‚0 network.recv‚568618 disk.sda.out‚40.09375î(2 - _wandbruntime‚1706  _runtime‚1706²\ ^Æ‚ À trainer/global_step‚3599 $ _timestamp‚1.784369764737656e+09 _runtime‚1706.183546293 _step‚288 & lr-AdamW/pg1‚0.0008374708902202691 & lr-AdamW/pg2‚0.0008374708902202691û¥—¥Ëƒ Å $ _timestamp‚1.784369764737656e+09 _runtime‚1706.183546293 _step‚288 & lr-AdamW/pg1‚0.0008374708902202691 & lr-AdamW/pg2‚0.0008374708902202691 trainer/global_step‚3599 ‚wƒÁ2„ - _wandbruntime‚1706  _runtime‚1706”Ë•§Î'… ÈO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚3599 _runtime‚1706.185459648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2118267085312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139570362495488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16562165 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137248113351622 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137452112449536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135137308444672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110891044040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34115858 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34125835 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137452095410176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131544288842752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139570467471872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137452095410176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8157525 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24719467 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128c2zËÓ'† ÍO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137452095410176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131544288842752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139570467471872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137452095410176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8157525 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24719467 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚3599 _runtime‚1706.185459648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2118267085312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139570362495488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16562165 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137248113351622 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137452112449536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135137308444672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110891044040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34115858 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34125835¡ö‹2‡ - _wandbruntime‚1706  _runtime‚1706EñÝkˆ ÑM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139608147802624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2118924388864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2118836444672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61070019 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26944882 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34135130 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131579788091392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26944880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129389950099456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137489206374400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137489206374400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24726036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16566649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61080012 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚1706.524214166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8159387 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61080012 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139608042819072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139608147802624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26944880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2189837991936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137285152708480 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34135130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34125139Fo]Ü&‰ ÖM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2118924388864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24726258 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131579586890752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2111458365682 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61070019 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26944882 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2118836444672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137489223413760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784369765080158e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8159606 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137285255885042 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2111372228480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚1706.524214166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8159387 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61080012 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139608042819072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139608147802624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26944880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2189837991936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137285152708480 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34135130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34125139 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139608147802624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2118924388864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2118836444672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61070019 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26944882 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34135130 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131579788091392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26944880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129389950099456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137489206374400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137489206374400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24726036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16566649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61080012 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25052879 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8266514 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139307643621376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139100216274180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27298709 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61890448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133319113029120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139307660660736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131100614316032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27298711 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139307643621376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27298711 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136960910211072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2139289023748 trainer/global_step‚3649 % _timestamp‚1.7843697874507453e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34581760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34591737 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27298709 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2146855247360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8266731 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2218693629440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131100589412352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34591737 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34581760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ ¡Üíh6845104128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61880469 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141454410931712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2146767310336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133319307945472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2139202886658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141454410931712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ý:ú’Ó'– ÍO _step‚293 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8266731 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2218693629440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131100589412352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34591737 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34581760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61880469 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141454410931712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2146767310336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133319307945472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2139202886658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141454410931712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2146767310336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25052879 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8266514 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139307643621376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139100216274180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27298709 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61890448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133319113029120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139307660660736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131100614316032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27298711 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139307643621376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27298711 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136960910211072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2139289023748 trainer/global_step‚3649 % _timestamp‚1.7843697874507453e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34581760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34591737 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27298709 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2146855247360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27305930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚141492091255296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16790849 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141492196238848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34601032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131135525830656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34591041 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25059537 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139344754585600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8268905 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136997399285760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133354616520192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34591041 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139137152454588 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61906964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2147424613888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61896971 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2219283501568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2219115593216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136997382246400 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2147336669696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8268688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61906964 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚141492196238848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25059757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16790852 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2139856345390 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327á0XÜ&™ ÖM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8268905 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136997399285760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133354616520192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34591041 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139137152454588 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61906964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2147424613888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61896971 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2219283501568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2219115593216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136997382246400 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2147336669696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8268688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61906964 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚141492196238848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25059757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16790852 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2139856345390 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2147336669696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139344771624960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2147424613888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139137255631150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27305930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚141492091255296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16790849 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141492196238848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34601032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131135525830656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34591041 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25059537 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139344754585600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2247982653952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211693568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35047662 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35047662 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35057639 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25386881 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143338564344320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186789888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2167687003456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2175267535360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚3699 _step‚297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143338564344320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141163191832576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27659759 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135094359491072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2248169443840 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138784529016832 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8376552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140952112843838 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138784511977472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132846190047232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141163191832576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62717400 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143338459367936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62707421 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35057639 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0e3Ó'¤ ÍO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141163208871936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8376316 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2175267535360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17010565 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141163208871936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27659761 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25387120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132846165143552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚677 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8376552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140952112843838 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138784511977472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚450 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132846190047232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141163191832576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62717400 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143338459367936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62707421 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35057639 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚140952216020288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2247982653952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211693568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35047662 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35047662 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35057639 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25386881 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143338564344320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186789888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2167687003456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2175267535360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚3699 _step‚297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143338564344320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141163191832576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27659759 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135094359491072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2248169443840 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138784529016832 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27666980 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27666980 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35056943 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141200319836160 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25393559 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141200302796800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143376139691520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62723923 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62723923 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27666982 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2248568023552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚1751.737484587 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2175924838912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62733916 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132881101561856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 _step‚298 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143376244675072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143376244675072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 " _timestamp‚1.7843698102934e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27666982 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2175836894720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2175836894720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138820984012800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÄùµÚ&§ ÔM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25393781 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 trainer/global_step‚3699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27666982 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2175836894720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2175836894720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚450 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138820984012800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚677 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27666980 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27666980 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35056943 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141200319836160 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25393559 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141200302796800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143376139691520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62723923 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62723923 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27666982 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2248568023552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚1751.737484587 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2175924838912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62733916 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132881101561856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 _step‚298 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143376244675072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143376244675072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 " _timestamp‚1.7843698102934e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17234768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145222612780544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145222507804160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8484933 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2277349413376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚3749 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143018757083136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2277521523200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28020811 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25719918 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25719698 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2195998846074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28020811 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1774.066826764 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134591740874752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140608113743872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35513564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28020809 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136869090288128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142804215766396 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63534373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28020809 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136869287301632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143018740043776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2203767760384 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2196084983164 % _timestamp‚1.7843698326228542e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35523541 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63544352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35513564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17234765 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134591765778432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _step‚301 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143018740043776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17234768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145222612780544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145222507804160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8484933 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2277349413376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚3749 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143018757083136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2277521523200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28020811 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25719918 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25719698 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2195998846074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28020811 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1774.066826764 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134591740874752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140608113743872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35513564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28020809 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136869090288128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142804215766396 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63534373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28020809 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136869287301632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143018740043776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2203767760384 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2196084983164 % _timestamp‚1.7843698326228542e+09 PCDeviceStatsMonitor.on_train_batch_start/act’ÞIßñ ive.small_pool.allocated‚35523541 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63544352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35513564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17234765 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134591765778432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _step‚301 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143018740043776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 B2µ - _wandbruntime‚1774  _runtime‚1774-ÉÍØ&¶ ÒM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136904586168832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚3749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2204425063936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28028030 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2278116367872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2196652304806 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25726394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134626652389376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17239249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143055851008000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63550875 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28028032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63560868 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚145260293111296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143055851008000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8487145 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚140644602818560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35522845 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63560868 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2204425063936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142841151946804 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35532836 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143055868047360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064w(Ý&· ×M H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143055851008000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8487145 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚140644602818560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35522845 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63560868 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2204425063936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142841151946804 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35532836 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143055868047360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136904586168832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚3749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2204425063936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28028030 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2278116367872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2196652304806 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25726394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134626652389376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17239249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143055851008000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63550875 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28028032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63560868 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚145260293111296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0®êÉ@@2¸ - _wandbruntime‚1774  _runtime‚1774ô ¹  /loss/FBt2/detection_loss‚0.4128182129296538 %loss/total_loss‚9.332969820169595 _runtime‚ 1774.41430099 2loss/SRqqt2/assignment_loss‚0.5213836155016907 /loss/FRt1/detection_loss‚0.5534801270186575 trainer/global_step‚3749 % _timestamp‚1.7843698329707646e+09 0loss/FRt2/assignment_loss‚1.1658680788317666 1loss/FBt1/assignment_loss‚0.10192908544813516 1loss/SRqqt1/detection_loss‚1.0954005647477487 1loss/SRqqt2/detection_loss‚0.7319283840285303 /loss/FBt1/detection_loss‚0.6985909732488486 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7058754282140445 3loss/FBt2/assignment_loss‚0.0022866648726955004 .loss/FRt2/detection_loss‚1.368936528925036 _step‚303 0loss/FRt1/assignment_loss‚1.9744721564027892´S‘ º † % _timestamp‚1.7843698329707646e+09 0loss/FRt2/assignment_loss‚1.1658680788317666 1loss/FBt1/assignment_loss‚0.10192908544813516 1loss/SRqqt1/detection_loss‚1.0954005647477487 1loss/SRqqt2/detection_loss‚0.7319283840285303 /loss/FBt1/detection_loss‚0.6985909732488486 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7058754282140445 3loss/FBt2/assignment_loss‚0.0022866648726955004 .loss/FRt2/detection_loss‚1.368936528925036 _step‚303 0loss/FRt1/assignment_loss‚1.9744721564027892 /loss/FBt2/detection_loss‚0.4128182129296538 %loss/total_loss‚9.332969820169595 _runtime‚ 1774.41430099 2loss/SRqqt2/assignment_loss‚0.5213836155016907 /loss/FRt1/detection_loss‚0.5534801270186575 trainer/global_step‚3749¯‘º¹œ» ‚(:‘ ³¥íқМŠ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚185.049' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.009800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S 4Ó¼ ‚(:È ³¥íÒÓñÛ$ proc.memory.availableMB‚132.9375# proc.memory.rssMB‚ 2691.11328125 proc.cpu.threads‚17 disk.sda1.in‚0! disk.dm-0.out‚4103.7216796875 disk.sda.out‚ 44.08203125 disk.dm-1.in‚ 0.10546875 disk.sda.in‚ 0.10546875* proc.memory.percent‚16.425251960754395# disk./.usageGB‚590.284065246582 disk.sda2.in‚0 network.sent‚ 4.868793e+06 network.recv‚596608" disk.nvme0c0n1.in‚ 2378.3515625 disk.rbd0.in‚0 cpu‚26.99035164106684 disk.sda1.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚39.61445890001796 disk.dm-1.out‚ 44.08203125 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚4103.7216796875 disk.nvme0n1.in‚ 2378.3515625$ disk.nvme0n1.out‚4103.7216796875 disk.rbd0.out‚0 disk.dm-0.in‚ 2378.3515625 disk.sda3.out‚ 44.08203125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.10546875 disk.rbd1.out‚4.375$ memory_percent‚99.18861389160156èfgö2½ - _wandbruntime‚1796  _runtime‚1796ðç²rƾ À & lr-AdamW/pg1‚0.0008359103982097984 & lr-AdamW/pg2‚0.0008359103982097984 trainer/global_step‚3799 $ _timestamp‚1.784369855159669e+09 _runtime‚1796.603519881 _step‚304€-w¢Ë¿ Å & lr-AdamW/pg1‚0.0008359103982097984 & lr-AdamW/pg2‚0.0008359103982097984 trainer/global_step‚3799 $ _timestamp‚1.784369855159669e+09 _runtime‚1796.603519881 _step‚304°›‹öõ2À - _wandbruntime‚1796  _runtime‚1796X+Ð%Á ÆO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147106556240384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35989443 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144656215512504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2232267985408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144874305294336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144874288254976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚136337341509632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147106661216768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142431715510272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2224482962872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2224396825782 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2232267985408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17458965 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64361325 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35989443 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28381859 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0…3]Ñ' ËO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64361325 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35989443 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28381859 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147106556240384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35989443 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144656215512504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2232267985408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144874305294336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144874288254976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚136337341509632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147106661216768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142431715510272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2224482962872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2224396825782 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2232267985408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17458965 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327±ïüo¿2à - _wandbruntime‚1796  _runtime‚1796çlûØ&Ä ÒM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2232837344768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213783552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28389082 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28389080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64377827 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147144341547520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64387820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2232837344768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144911399219200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35998738 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28389082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35998738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1796.945629102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17463452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚3799 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2224964147312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚147144236563968 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8596174 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28389080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147144341547520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8595951 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17463449 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2307364145152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26059400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26059626 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144911416258560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142468204584960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2232925288960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64387820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144911416258560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2232925288960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136372253024256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138679806049280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138679592265728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2307553025024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144693254869474 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144693151692912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144911399219200 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35998738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1796.945629102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17463452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚3799 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2224964147312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚147144236563968 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8596174 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28389080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147144341547520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8595951 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17463449 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2307364145152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26059400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26059626 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144911416258560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142468204584960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2232925288960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64387820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144911416258560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2232925288960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136372253024256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138679806049280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138679592265728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2307553025024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144693254869474 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144693151692912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144911399219200 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148990709652992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2336357404672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2260856147456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28742911 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146729836466176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2260768210432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148990604676608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148990604676608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146729853505536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28742911 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140419069243392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144255317276672 % _timestamp‚1.7843698773092098e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8703142 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36445368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146729853505536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17683168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473f(™yÓ'Ò ÍO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146729853505536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28742911 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140419069243392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144255317276672 % _timestamp‚1.7843698773092098e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8703142 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36445368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146729853505536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17683168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36455345 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148990709652992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2336357404672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2260856147456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28742911 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146729836466176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2260768210432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148990604676608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148990604676608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0µ!‘ñ2Ó - _wandbruntime‚1819  _runtime‚1819xñ‘x'Ô ÒM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138117803851776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2336762791936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253448264222 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144291806351360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28750132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65214772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36454649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65214772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65204779 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36454649 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149028389983744 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28750130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8705378 _step‚310 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149028285000192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146766964469760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17687652 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8705611 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146766947430400 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚1819.097915031 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36464640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65204779 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144291789312000 trainer/global_step‚3849 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36464640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2336941186048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253362127020 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146766947430400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end Ù´Þ± /allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17687649 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843698776538181e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2261337569792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28750132 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146545151439020 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26393027 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26393263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2261425513984 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146766947430400 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚1819.097915031 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36464640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65204779 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144291789312000 trainer/global_step‚3849 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36464640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2336941186048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253362127020 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146766947430400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17687649 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843698776538181e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2261337569792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28750132 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146545151439020 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26393027 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26393263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2261425513984 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144291806351360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28750132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65214772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36454649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65214772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65204779 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36454649 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149028389983744 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28750130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8705378 _step‚310¶½¶.2Ö - _wandbruntime‚1819  _runtime‚1819:[ˆ× € 2loss/SRqqt1/assignment_loss‚0.6429267593267713 0loss/FBt1/assignment_loss‚0.1669721315964339 /loss/FRt1/detection_loss‚0.6278812071083338 /loss/FRt2/detection_loss‚1.3831904020800951 1loss/SRqqt1/detection_loss‚0.9541735700356266 /loss/FBt1/detection_loss‚1.0021060023675947 0loss/FRt2/assignment_loss‚1.0644124289275159 trainer/global_step‚3849 _step‚311 0loss/FRt1/assignment_loss‚1.8634033153762974 % _timestamp‚1.7843698776559696e+09 _runtime‚1819.099507829 2loss/SRqqt2/assignment_loss‚0.6265719257495352 2loss/FBt2/assignment_loss‚0.048876435646396474 0loss/SRqqt2/detection_loss‚0.717680139122082 /loss/FBt2/detection_loss‚0.4644556132434691 %loss/total_loss‚9.562649930580152 epoch‚4 Ô௠Ø … 0loss/FRt1/assignment_loss‚1.8634033153762974 % _timestamp‚1.7843698776559696e+09 _runtime‚1819.099507829 2loss/SRqqt2/assignment_loss‚0.6265719257495352 2loss/FBt2/assignment_loss‚0.048876435646396474 0loss/SRqqt2/detection_loss‚0.717680139122082 /loss/FBt2/detection_loss‚0.4644556132434691 %loss/total_loss‚9.562649930580152 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6429267593267713 0loss/FBt1/assignment_loss‚0.1669721315964339 /loss/FRt1/detection_loss‚0.6278812071083338 /loss/FRt2/detection_loss‚1.3831904020800951 1loss/SRqqt1/detection_loss‚0.9541735700356266 /loss/FBt1/detection_loss‚1.0021060023675947 0loss/FRt2/assignment_loss‚1.0644124289275159 trainer/global_step‚3849 _step‚311·o0çÙ ‚(:… à¥íÒد‹ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚154.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;;ÁºÒÚ ‚(:Ç à¥íÒÑΡŒ# disk.nvme0n1.out‚4115.259765625# proc.memory.rssMB‚ 2691.11328125 cpu‚26.981208752152675 network.sent‚ 4.998285e+06# disk./.usageGB‚590.283992767334 disk.sda3.out‚ 44.82421875 disk.sda.in‚ 0.10546875 disk.nvme0n1.in‚ 2513.4453125* proc.memory.percent‚16.425251960754395 disk.sda2.in‚0 disk.rbd1.out‚4.5 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 133.2109375 network.recv‚609382$ memory_percent‚99.18694496154785) disk./.usagePercent‚39.61445403587481 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚ 44.82421875% disk.nvme0c0n1.out‚4115.259765625 disk.sda1.in‚0 disk.dm-0.in‚ 2513.4453125 disk.dm-0.out‚4115.259765625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 44.82421875" disk.nvme0c0n1.in‚ 2513.4453125ÚÎ!2Û - _wandbruntime‚1840  _runtime‚1840]®²ÅÜ ¿ _runtime‚1840.541326741 _step‚312 & lr-AdamW/pg1‚0.0008350905443762032 & lr-AdamW/pg2‚0.0008350905443762032 trainer/global_step‚3899 # _timestamp‚1.78436989909753e+09É]—ÓÊÝ Ä & lr-AdamW/pg1‚0.0008350905443762032 & lr-AdamW/pg2‚0.0008350905443762032 trainer/global_step‚3899 # _timestamp‚1.78436989909753e+09 _runtime‚1840.541326741 _step‚312¸“Ceþ2Þ - _wandbruntime‚1840  _runtime‚1840ŠBÀ&Î'ß ÈO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2289268435456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2281192785198 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36921247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142194185426432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142193982121472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29103959 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148585401716736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148585401716736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139828468068352 _step‚313 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2289356372480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150874758089216 % _timestamp‚1.7843698990986764e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148585384677376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66015229 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26719992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26719759 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146078919043072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8812394 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8812624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17907368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146078936082432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150874653112832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148360111828270 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66025208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2289356372480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36911270 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36921247 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36911270 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1840.542718358 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148360215004720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139828492972032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150874653112832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29103961 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17907365 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29103959 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2281278922288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0@ñ^é±à ÍO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146078936082432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150874653112832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148360111828270 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66025208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2289356372480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36911270 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36921247 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36911270 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1840.542718358 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148360215004720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139828492972032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150874653112832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29103961 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17907365 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29103959 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2281278922288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2289268435456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2281192785198 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36921247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142194185426432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142193982121472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.sm~p·ö"all_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29103959 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148585401716736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148585401716736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139828468068352 _step‚313 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2289356372480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150874758089216 % _timestamp‚1.7843698990986764e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148585384677376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66015229 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26719992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26719759 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146078919043072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8812394 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8812624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17907368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146¹°Y72á - _wandbruntime‚1840  _runtime‚1840™ýØ&â ÒM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142229480447488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843698994391272e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148622512680960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚3899 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148397254361690 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66031731 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814602 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2366281355776 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29111180 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8814827 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139863404486656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148397151185128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚146115408117760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36920551 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150912438419968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2289925739008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1840.883166722 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66031731 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29111182 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139863379582976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26726679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148622512680960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2366100864512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26726451 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36920551 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148622495641600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66041724 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚146115391078400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150912333436416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29111182 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139863379582976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26726679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148622512680960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2366100864512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26726451 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36920551 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148622495641600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66041724 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚146115391078400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150912333436416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66031731 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814602 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2366281355776 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29111180 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8814827 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139863404486656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148397151185128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚146115408117760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36920551 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150912438419968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2289925739008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1840.883166722 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66031731 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936º,òè2ä - _wandbruntime‚1840  _runtime‚1840¼‰å< å ƒ 1loss/FBt2/assignment_loss‚0.02369380492830994 1loss/SRqqt2/detection_loss‚0.7404359978841836 /loss/FBt2/detection_loss‚0.6132003586312913 trainer/global_step‚3899 epoch‚4 2loss/SRqqt2/assignment_loss‚0.5667673302097693 /loss/FRt1/detection_loss‚0.5797962977501432 .loss/FRt2/detection_loss‚1.330076856580668 /loss/FBt1/detection_loss‚0.8749096989631653 _step‚315 0loss/FRt1/assignment_loss‚1.8357334372897942 1loss/SRqqt1/detection_loss‚1.1041686412622065 % _timestamp‚1.7843698994407985e+09 _runtime‚1840.884332623 %loss/total_loss‚9.538158039616263 0loss/FRt2/assignment_loss‚1.1521541814606888 2loss/SRqqt1/assignment_loss‚0.7167734330646606 4loss/FBt1/assignment_loss‚0.00044800159138134175As æ ˆ .loss/FRt2/detection_loss‚1.330076856580668 /loss/FBt1/detection_loss‚0.8749096989631653 _step‚315 0loss/FRt1/assignment_loss‚1.8357334372897942 1loss/SRqqt1/detection_loss‚1.1041686412622065 % _timestamp‚1.7843698994407985e+09 _runtime‚1840.884332623 %loss/total_loss‚9.538158039616263 0loss/FRt2/assignment_loss‚1.1521541814606888 2loss/SRqqt1/assignment_loss‚0.7167734330646606 4loss/FBt1/assignment_loss‚0.00044800159138134175 1loss/FBt2/assignment_loss‚0.02369380492830994 1loss/SRqqt2/detection_loss‚0.7404359978841836 /loss/FBt2/detection_loss‚0.6132003586312913 trainer/global_step‚3899 epoch‚4 2loss/SRqqt2/assignment_loss‚0.5667673302097693 /loss/FRt1/detection_loss‚0.5797962977501432»JÆ;‹‘ç ‚(:† ï¥íұñŠ gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¤ Ðè ‚(:Å ï¥íÒü²ÂŒ disk.sda1.out‚0 disk.dm-1.out‚44.90625 disk.rbd1.in‚0 disk.sda.out‚44.90625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2691.11328125" disk.dm-0.out‚4120.71142578125( proc.memory.availableMB‚ 132.71484375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 2513.4453125 disk.rbd1.out‚4.5 disk.sda.in‚ 0.10546875" disk.nvme0c0n1.in‚ 2513.4453125) disk./.usagePercent‚39.61445121979193 disk.sda3.in‚ 0.10546875 disk.sda3.out‚44.90625 disk.drbd2165.out‚0% disk.nvme0n1.out‚4120.71142578125* proc.memory.percent‚16.425251960754395 disk.sda1.in‚0$ memory_percent‚99.18997287750244 cpu‚26.97547769205304 disk.sda2.out‚0' disk.nvme0c0n1.out‚4120.71142578125 network.recv‚614630 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 2513.4453125 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 5.060864e+06$ disk./.usageGB‚590.2839508056641n\°£é ‚(:… þ¥íÒæ•‹ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150212214750828 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150212111574378 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37377172 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2317768660480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141574068703232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152758701549056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150440932888576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2317768660480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 Et<ÅÒF=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚1862.458902745 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66852160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37387149 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27053134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29465009 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141574043799552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37387149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18131568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2309676901996 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147902520809472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2317856597504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150440949927936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27052914 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18131565 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3949 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2309590764906 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29465011 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144AGÒ'ð ÌO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152758701549056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150440932888576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2317768660480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚1862.458902745 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66852160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37387149 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27053134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29465009 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141574043799552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37387149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18131568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2309676901996 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147902520809472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2317856597504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150440949927936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27052914 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18131565 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3949 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2309590764906 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29465011 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150212214750828 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150212111574378 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37377172 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2317768660480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141574068703232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66868676 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66858683 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37386453 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29472230 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152796486856192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147938992844800 _runtime‚1862.802773368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66858683 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150478060892160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150478060892160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150249150931236 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29472232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18136049 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144004484341248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2310158086436 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66868676 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150478043852800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2318425964032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144004689736192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2310244223638 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18136052 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141608955314176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚318 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152796381872640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843699213586645e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37386453 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150478043852800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37396444 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8923529 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27059578 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29472230²¡»ó ×M T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152796486856192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147938992844800 _runtime‚1862.802773368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66858683 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150478060892160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150478060892160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150249150931236 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29472232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18136049 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144004484341248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2310158086436 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66868676 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150478043852800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2318425964032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144004689736192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2310244223638 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18136052 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141608955314176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚318 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152796381872640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843699213586645e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37386453 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStaØŽ*í"tsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150478043852800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37396444 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8923529 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27059578 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29472230 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2318338019840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27059818 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141608980217856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37396444 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2395709518336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2318338019840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2318425964032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147939009884160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66868676 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66858683 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37386453 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29472230 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¾¼‚2ô - _wandbruntime‚1862  _runtime‚1862zúVõ þ /loss/FRt1/assignment_loss‚2.039178740132079 2loss/SRqqt2/assignment_loss‚0.4824779406112515 /loss/FRt1/detection_loss‚0.6022965008039644 /loss/FRt2/detection_loss‚1.2464733681208302 /loss/FBt2/detection_loss‚0.6157980206750195 $loss/total_loss‚9.12266402755844 epoch‚4 2loss/FBt2/assignment_loss‚0.007762936695262191 /loss/FBt1/detection_loss‚0.5079092658124864 trainer/global_step‚3949 0loss/FRt2/assignment_loss‚0.9972376654871175 2loss/SRqqt1/assignment_loss‚0.7054306844308396 1loss/FBt1/assignment_loss‚0.02092180955288825 1loss/SRqqt2/detection_loss‚0.6528248455588448 _runtime‚1862.804064396 0loss/SRqqt1/detection_loss‚1.244352249677857 $ _timestamp‚1.784369921360534e+09 _step‚319]   ö ƒ 0loss/SRqqt1/detection_loss‚1.244352249677857 $ _timestamp‚1.784369921360534e+09 _step‚319 /loss/FRt1/assignment_loss‚2.039178740132079 2loss/SRqqt2/assignment_loss‚0.4824779406112515 /loss/FRt1/detection_loss‚0.6022965008039644 /loss/FRt2/detection_loss‚1.2464733681208302 /loss/FBt2/detection_loss‚0.6157980206750195 $loss/total_loss‚9.12266402755844 epoch‚4 2loss/FBt2/assignment_loss‚0.007762936695262191 /loss/FBt1/detection_loss‚0.5079092658124864 trainer/global_step‚3949 0loss/FRt2/assignment_loss‚0.9972376654871175 2loss/SRqqt1/assignment_loss‚0.7054306844308396 1loss/FBt1/assignment_loss‚0.02092180955288825 1loss/SRqqt2/detection_loss‚0.6528248455588448 _runtime‚1862.804064396¿Ü‘÷ ‚(:† ¦íÒˆ¦ùŠ gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚173.108' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6216 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ– ¶Üø ‚(:Ñ ¦íÒ鹚 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚ 45.48046875 disk.dm-0.in‚ 2594.4609375 disk.sda.in‚ 0.10546875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 2594.4609375 disk.nvme0n1.in‚ 2594.4609375 disk.rbd0.in‚0 network.sent‚ 5.128625e+06$ disk./.usageGB‚590.2839508056641" disk.dm-0.out‚4129.33251953125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚99.18942451477051 disk.rbd1.out‚4.5625 disk.sda2.in‚0' proc.memory.availableMB‚ 132.8046875 disk.sda3.out‚ 45.48046875 disk.drbd2165.out‚0% disk.nvme0n1.out‚4129.33251953125* proc.memory.percent‚16.425251960754395 cpu‚26.964018529111517) disk./.usagePercent‚39.61445121979193 disk.sda.out‚ 45.48046875# proc.memory.rssMB‚ 2691.11328125' disk.nvme0c0n1.out‚4129.33251953125 proc.cpu.threads‚17 network.recv‚622971 disk.sda3.in‚ 0.10546875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0RQ?Ü2ù - _wandbruntime‚1884  _runtime‚1884 S!Çú Á % _timestamp‚1.7843699429980881e+09 _runtime‚1884.441901857 _step‚320 & lr-AdamW/pg1‚0.0008342443547951983 & lr-AdamW/pg2‚0.0008342443547951983 trainer/global_step‚3999zQ¡€Ìû Æ _step‚320 & lr-AdamW/pg1‚0.0008342443547951983 & lr-AdamW/pg2‚0.0008342443547951983 trainer/global_step‚3999 % _timestamp‚1.7843699429980881e+09 _runtime‚1884.441901857ÀÎÈ”2ü - _wandbruntime‚1884  _runtime‚1884Ã.Z_Í'ý ÇO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2346356822528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27385611 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9029846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67669133 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18355765 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9030062 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145744004465664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152064214496936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149726139615232 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152296498139136 $ _timestamp‚1.784369942999153e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143319644434432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152296481099776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145744203576320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 _step‚321 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154642749985280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152296498139136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29826059 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154642854961664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2346356822528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29826059 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2346268885504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29826061 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1884.443114866 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154642854961664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152296481099776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29826061 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143319619530752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149726122575872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2424384934912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2346268885504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27385830 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2337988744614 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67679112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2338074881704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37853051 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37853051 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37843074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2424559141888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154642749985280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37853051 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37843074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2424559141888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154642749985280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145744004465664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152064214496936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149726139615232 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152296498139136 $ _timestamp‚1.784369942999153e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143319644434432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152296481099776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145744203576320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 _step‚321 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154642749985280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152296498139136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29826059 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154642854961664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2346356822528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29826059 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2346268885504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29826061 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1884.443114866 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154642854961664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152296481099776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29826061 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143319619530752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149726122575872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2424384934912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2346268885504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27385830 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2337988744614 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67679112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2338074881704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37853051 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9032194 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29833280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67695628 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1884.782785155 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.smalw”ÎÚÎl_pool.freed‚ 2338556066144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152101253853906 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27392225 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67685635 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2346838244864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2338642203346 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37862346 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152333609103360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2346926189056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145779498392576 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29833282 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27392446 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152101150677344 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154680430308864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152333592064000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143354555949056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149762611650560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67685635 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 trainer/global_step‚3999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37852355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9031976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37852355 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2346926189056«]øœÛ&ÕM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154680430308864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152333592064000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143354555949056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149762611650560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67685635 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 trainer/global_step‚3999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37852355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9031976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37852355 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2346926189056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29833282 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2346838244864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18360252 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143354531045376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2425149935616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9032194 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29833280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67695628 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1884.782785155 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2338556066144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152101253853906 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27392225 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67685635 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2346838244864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2338642203346 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37862346 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152333609103360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2346926189056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145779498392576 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29833282 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27392446 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152101150677344­gN 2‚- _wandbruntime‚1884  _runtime‚1884f Iiƒ 2loss/FBt1/assignment_loss‚0.005890162934368466 /loss/FRt1/detection_loss‚0.6355389330214954 epoch‚4 0loss/FRt2/assignment_loss‚1.0117488572763842 2loss/SRqqt1/assignment_loss‚0.8071322842912382 /loss/FRt2/detection_loss‚1.2109049979070303 %loss/total_loss‚9.390021573016668 % _timestamp‚1.7843699433403478e+09 0loss/FRt1/assignment_loss‚1.8979969043016924 2loss/SRqqt2/assignment_loss‚0.5986430867270346 1loss/SRqqt2/detection_loss‚0.6607643136204305 _runtime‚1884.783883478 _step‚323 2loss/FBt2/assignment_loss‚0.011115433971211508 /loss/SRqqt1/detection_loss‚1.24291917255461 /loss/FBt1/detection_loss‚0.7154089162747065 /loss/FBt2/detection_loss‚0.5919585101364646 trainer/global_step‚3999Ûû|® „† 2loss/FBt1/assignment_loss‚0.005890162934368466 /loss/FRt1/detection_loss‚0.6355389330214954 epoch‚4 0loss/FRt2/assignment_loss‚1.0117488572763842 2loss/SRqqt1/assignment_loss‚0.8071322842912382 /loss/FRt2/detection_loss‚1.2109049979070303 %loss/total_loss‚9.390021573016668 % _timestamp‚1.7843699433403478e+09 0loss/FRt1/assignment_loss‚1.8979969043016924 2loss/SRqqt2/assignment_loss‚0.5986430867270346 1loss/SRqqt2/detection_loss‚0.6607643136204305 _runtime‚1884.783883478 _step‚323 2loss/FBt2/assignment_loss‚0.011115433971211508 /loss/SRqqt1/detection_loss‚1.24291917255461 /loss/FBt1/detection_loss‚0.7154089162747065 /loss/FBt2/detection_loss‚0.5919585101364646 trainer/global_step‚3999Ûñ*‡‘…‚(:† œ¦íÒúüüŠ gpu.0.gpu‚53.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öb ׆‚(:Ì œ¦íÒ¢ïúŒ disk.sda1.in‚0$ memory_percent‚99.19469356536865( proc.memory.availableMB‚ 131.94140625) disk./.usagePercent‚39.61442689907618 disk.rbd1.out‚4.625" disk.nvme0c0n1.in‚ 2594.4609375" proc.memory.rssMB‚ 2691.7734375 disk.dm-1.out‚ 45.67578125 disk.sda.out‚ 45.67578125 disk.nvme0n1.in‚ 2594.4609375$ disk.nvme0n1.out‚4133.1591796875 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 2594.4609375 disk.sda3.out‚ 45.67578125 disk.sda.in‚ 0.10546875 disk.drbd2165.out‚0 disk.rbd1.in‚0) proc.memory.percent‚16.42928123474121 disk.rbd0.out‚0 network.sent‚ 5.191395e+06 disk.sda2.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚4133.1591796875 disk.rbd0.in‚0 cpu‚26.964958853306648 network.recv‚628019 disk.sda2.out‚0 disk.sda3.in‚ 0.10546875 proc.cpu.threads‚17$ disk./.usageGB‚590.2835884094238! disk.dm-0.out‚4133.1591796875 disk.sda1.out‚0Ž'~¤š‡‚(: «¦íÒáÒÉŠ gpu.0.gpu‚68.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚137.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.560000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X ‚{ψ‚(:Ä «¦íÒÜ„èŒ% disk.nvme0c0n1.out‚4139.716796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.10546875 cpu‚26.990976700409114 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 45.953125 disk.sda3.in‚ 0.10546875* proc.memory.percent‚16.449427604675293 disk.rbd1.out‚4.625 disk.rbd0.out‚0 disk.sda3.out‚ 45.953125" memory_percent‚99.214768409729# proc.memory.rssMB‚ 2695.07421875 network.sent‚ 5.253915e+06$ disk./.usageGB‚590.2840614318848" disk.nvme0c0n1.in‚ 2594.4609375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚ 45.953125* disk./.usagePercent‚39.614458644010426 disk.drbd2165.out‚0# disk.nvme0n1.out‚4139.716796875 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 2594.4609375 network.recv‚633203 disk.nvme0n1.in‚ 2594.4609375 disk.dm-0.out‚4139.716796875 disk.drbd2165.in‚0( proc.memory.availableMB‚ 128.65234375îç182‰- _wandbruntime‚1906  _runtime‚19067®FÂ1ŠÛ{ 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9924097657203674 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3970485329627991 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6341129541397095 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6192091107368469 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6017830967903137 *jet/accuracy_2_of_4‚0.7171436548233032 7validation_average_jet_accuracy‚0.31518951058387756 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547739386558533 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6017830967903137 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5992844700813293 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6088255047798157 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.2652353048324585 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6008296608924866 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13089391589164734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7736214995384216 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 _step‚324 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6044480204582214 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5802038908004761 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02728254161775112 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08322467654943466 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15228378772735596 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09620148688554764 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08698846399784088 +jet/accuracy_3_of_3‚0.07003184407949448 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6726398468017578 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08322467654943466 jet/accuracy_2_of_5‚1 5 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.99720299243927 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08038665354251862 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4548475742340088 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996731698513031 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7285078167915344 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7474358081817627 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10528337210416794 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981008172035217 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34830164909362793 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5573395490646362 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10221078246831894 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09184916317462921 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6336938142776489 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0987524762749672 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6743146181106567 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009470389224588871 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950737953186035 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8929271697998047 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01661774516105652 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6466591954231262 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6754307150840759 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2930503785610199 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02853999473154545 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12989594042301178 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6504212021827698 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1743108034133911 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08497621864080429 (particle/accuracy‚0.8439863920211792 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882401823997498 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3860079348087311 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10901539772748947 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6132275462150574 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7032822370529175 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.616589367389679 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10261797904968262 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9153631925582886 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.4989335536956787 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11025292426347733 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12498269975185394 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7484943270683289 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08934259414672852 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08360821008682251 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3641975224018097 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08698846399784088 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6379895210266113 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8500000238418579 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19939467310905457 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06689415872097015 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9936224818229675 *particle/specificity‚0.906484067440033 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6727578043937683 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,jet/accuracy_4_of_4‚0.014682539738714695 /particle/accuracy_2_of_2‚0.9998950958251953 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7539473176002502 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 $ _timestamp‚1.784369964617389e+09 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938385486602783 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937705993652344 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 *jet/accuracy_1_of_4‚0.9548366069793701 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7447724342346191 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6867520213127136 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.16661617159843445 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9837166666984558 .particle/accuracy_3_of_3‚0.999514102935791 epoch‚4 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44999998807907104 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6673280000686646 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5785372853279114 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4166666269302368 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47776898741722107 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004832011181861162 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996731698513031 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7235145568847656 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7473747730255127 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.767105221748352 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8673042058944702 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985426127910614 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1019548699259758 particle/accuracy_1_of_2‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10008834302425385 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7484943270683289 *jet/accuracy_1_of_2‚0.7239620685577393 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5517959594726562 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1022702157497406 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6195321083068848 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6154894232749939 )loss/val_total_loss‚9.144113653904881 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9856447577476501 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6045870780944824 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7116711735725403 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5317708253860474 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6556872725486755 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24248163402080536 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957804083824158 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7064694166183472 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6721989512443542 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014507881365716457 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1776×2= 355504989624 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6367572546005249 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7266072034835815 'particle/f_score‚0.6288388967514038 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08800536394119263 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 *jet/accuracy_1_of_3‚0.8909379243850708 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09518362581729889 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5828505158424377 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952496886253357 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9117187261581421 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.9503392577171326 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 _runtime‚1906.061863151 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09500306099653244 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.970852255821228 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10418462753295898 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976144433021545 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47055763006210327 particle/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08700548112392426 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8500000238418579 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.13713254034519196 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6219907402992249 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4845433533191681 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10047873854637146 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6337239742279053 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6180555820465088 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35301944613456726 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 trainer/global_step‚4004 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6025135517120361ÌAÍæ=‹à{ 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6743146181106567 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009470389224588871 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950737953186035 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8929271697998047 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01661774516105652 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6466591954231262 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6754307150840759 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2930503785610199 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02853999473154545 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12989594042301178 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6504212021827698 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1743108034133911 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08497621864080429 (particle/accuracy‚0.8439863920211792 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882401823997498 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3860079348087311 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10901539772748947 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6132275462150574 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7032822370529175 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.616589367389679 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10261797904968262 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9153631925582886 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.4989335536956787 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11025292426347733 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12498269975185394 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7484943270683289 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08934259414672852 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08360821008682251 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3641975224018097 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08698846399784088 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6379895210266113 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8500000238418579 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19939467310905457 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06689415872097015 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9936224818229675 *particle/specificity‚0.906484067440033 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6727578043937683 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,jet/accuracy_4_of_4‚0.014682539738714695 /particle/accuracy_2_of_2‚0.9998950958251953 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7539473176002502 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 $ _timestamp‚1.784369964617389e+09 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938385486602783 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937705993652344 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 *jet/accuracy_1_of_4‚0.9548366069793701 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7447724342346191 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6867520213127136 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.16661617159843445 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9837166666984558 .particle/accuracy_3_of_3‚0.999514102935791 epoch‚4 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44999998807907104 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6673280000686646 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5785372853279114 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4166666269302368 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47776898741722107 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004832011181861162 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996731698513031 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7235145568847656 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7473747730255127 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.767105221748352 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8673042058944702 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985426127910614 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1019548699259758 particle/accuracy_1_of_2‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10008834302425385 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7484943270683289 *jet/accuracy_1_of_2‚0.7239620685577393 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5517959594726562 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1022702157497406 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6195321083068848 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6154894232749939 )loss/val_total_loss‚9.144113653904881 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9856447577476501 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6045870780944824 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7116711735725403 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5317708253860474 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6556872725486755 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24248163402080536 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957804083824158 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7064694166183472 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6721989512443542 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014507881365716457 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1776355504989624 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6367572546005249 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7266072034835815 'particle/f_score‚0.6288388967514038 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08800536394119263 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 *jet/accuracy_1_of_3‚0.8909379243850708 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09518362581729889 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5828505158424377 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952496886253357 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9117187261581421 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.9503392577171326 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 _runtime‚1906.061863151 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09500306099653244 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.970852255821228 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10418462753295898 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976144433021545 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47055763006210327 particle/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08700548112392426 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8500000238418579 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.13713254034519196 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6219907402992249 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4845433533191681 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10047873854637146 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6337239742279053 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6180555820465088 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35301944613456726 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 trainer/global_step‚4004 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6025135517120361 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9924097657203674 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3970485329627991 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6341129541397095 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6192091107368469 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6017830967903137 *jet/accuracy_2_of_4‚0.7171436548233032 7validation_average_jet_accuracy‚0.31518951058387756 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547739386558533 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6017830967903137 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5992844700813293 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6088255047798157 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.2652353048324585 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6008296608924866 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13089391589164734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7736214995384216 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 _step‚324 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6044480204582214 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5802038908004761 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02728254161775112 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08322467654943466 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15228378772735596 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09620148688554764 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08698846399784088 +jet/accuracy_3_of_3‚0.07003184407949448 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6726398468017578 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08322467654943466 jet/accuracy_2_of_5‚1 5 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.99720299243927 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08038665354251862 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4548475742340088 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996731698513031 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7285078167915344 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7474358081817627 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10528337210416794 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981008172035217 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34830164909362793 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5573395490646362 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10221078246831894 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09184916317462921 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6336938142776489 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0987524762749672 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904ÄÑÍqœŒ‚(:‘ º¦íÒÛœˆ‹ gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚179.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.919200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é?ÈÏÖ‚(:Ë º¦íÒò¢— network.sent‚ 5.259004e+06 disk.sda3.out‚ 46.2734375 disk.rbd0.in‚0 disk.dm-0.in‚ 2594.4609375 disk.sda3.in‚ 0.10546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.10546875( proc.memory.availableMB‚ 196.91015625) disk./.usagePercent‚39.61447682054535 disk.sda.in‚ 0.10546875 disk.nvme0n1.in‚ 2594.4609375 disk.drbd2165.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚4207.58935546875 proc.memory.rssMB‚ 2695.1875' disk.nvme0c0n1.out‚4207.58935546875$ memory_percent‚98.79815578460693 disk.sda.out‚ 46.2734375 disk.sda2.out‚0 disk.dm-1.out‚ 46.2734375" disk.dm-0.out‚4207.58935546875 cpu‚26.986286288993405$ disk./.usageGB‚590.2843322753906 disk.drbd2165.out‚0 disk.rbd1.out‚4.6875* proc.memory.percent‚16.450119018554688 network.recv‚636541" disk.nvme0c0n1.in‚ 2594.4609375 disk.rbd0.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17êE©ô2Ž- _wandbruntime‚1926  _runtime‚192616ÖVÆÀ % _timestamp‚1.7843699852763078e+09 _runtime‚ 1926.72015088 _step‚325 & lr-AdamW/pg1‚0.0008338114009753388 & lr-AdamW/pg2‚0.0008338114009753388 trainer/global_step‚4049HcKƒËÅ & lr-AdamW/pg1‚0.0008338114009753388 & lr-AdamW/pg2‚0.0008338114009753388 trainer/global_step‚4049 % _timestamp‚1.7843699852763078e+09 _runtime‚ 1926.72015088 _step‚325År5›×2‘- _wandbruntime‚1926  _runtime‚1926اpµÍ'’ÇO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157504291632640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157504396609024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚155113025703936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚155113008664576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2391282968064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145979708686336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154889817541960 _step‚326 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2391282968064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157504291632640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68884170 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148450324311552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38462531 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38452554 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18684548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2470414417408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2470590721536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154889920718410 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155113008664576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157504396609024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27903522 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚1926.721400809 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30421637 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38452554 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30421639 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9218748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68874191 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18684545 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30421637 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9218974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152506959255552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101Ρ_“ÌO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27903293 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145979733590016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2391370905088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.currÁ¼†Q!ent‚226 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚155113025703936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38462531 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68874191 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18684545 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30421637 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9218974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152506959255552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157504291632640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157504396609024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚155113025703936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚155113008664576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2391282968064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145979708686336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154889817541960 _step‚326 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2391282968064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157504291632640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68884170 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148450324311552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38462531 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38452554 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18684548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2470414417408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2470590721536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154889920718410 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155113008664576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157504396609024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27903522 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚1926.721400809 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30421637 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38452554 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30421639 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9218748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 B§P2”- _wandbruntime‚1927  _runtime‚1927k´é–Ø&•ÒM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30428860 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2383442647298 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38461835 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146014620200960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2471007245824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38471826 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2391940271616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391852327424 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157541971956224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68900686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154926960075380 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157542076939776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157542076939776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚1927.064491195 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2383528784500 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27909934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18689029 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30428858 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9220905 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30428858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27910158 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68900686 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155150119628800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2391940271616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391852327424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68890693 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148485822355968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2471177251328 trainer/global_step‚4049 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30428860 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2383442647298 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38461835 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146014620200960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2471007245824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38471826 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2391940271616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391852327424 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157541971956224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68900686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154926960075380 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157542076939776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157542076939776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚1927.064491195 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2383528784500 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27909934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18689029 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30428858 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9220905 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30428858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27910158 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68900686 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155150119628800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2391940271616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391852327424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68890693 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148485822355968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2471177251328 trainer/global_step‚4049ÇJÐÉP2—- _wandbruntime‚1927  _runtime‚1927`îr¢˜‚ 0loss/FRt1/assignment_loss‚1.8451492660413913 0loss/FRt2/assignment_loss‚0.8575040671929895 .loss/FBt2/detection_loss‚0.597410935367788 trainer/global_step‚4049 1loss/SRqqt2/detection_loss‚0.5592514767217364 %loss/total_loss‚8.663336045914878 _runtime‚1927.065649421 _step‚328 epoch‚5 2loss/SRqqt1/assignment_loss‚0.6843673816562136 3loss/FBt2/assignment_loss‚0.0007811367155779046 /loss/FRt2/detection_loss‚1.3770979600299922 1loss/SRqqt1/detection_loss‚0.9531392107320416 .loss/FBt1/detection_loss‚0.644808234879747 % _timestamp‚1.7843699856220999e+09 2loss/SRqqt2/assignment_loss‚0.5533967531084839 2loss/FBt1/assignment_loss‚0.009807820844088155 /loss/FRt1/detection_loss‚0.5806218026248281Û‰ýY ™‡ 2loss/SRqqt2/assignment_loss‚0.5533967531084839 2loss/FBt1/assignment_loss‚0.009807820844088155 /loss/FRt1/detection_loss‚0.5806218026248281 0loss/FRt1/assignment_loss‚1.8451492660413913 0loss/FRt2/assignment_loss‚0.8575040671929895 .loss/FBt2/detection_loss‚0.597410935367788 trainer/global_step‚4049 1loss/SRqqt2/detection_loss‚0.5592514767217364 %loss/total_loss‚8.663336045914878 _runtime‚1927.065649421 _step‚328 epoch‚5 2loss/SRqqt1/assignment_loss‚0.6843673816562136 3loss/FBt2/assignment_loss‚0.0007811367155779046 /loss/FRt2/detection_loss‚1.3770979600299922 1loss/SRqqt1/detection_loss‚0.9531392107320416 .loss/FBt1/detection_loss‚0.644808234879747 % _timestamp‚1.7843699856220999e+09ȼRnD‘š‚(:† ɦíÒôʤ‹ gpu.0.gpu‚49.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚176.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FðPÑ›‚(:Æ ɦíÒŒ¥ cpu‚26.980528558386098$ disk./.usageGB‚590.2836151123047& disk.nvme0c0n1.out‚4278.3232421875 proc.memory.rssMB‚ 2695.1875* proc.memory.percent‚16.450119018554688 disk.sda.in‚ 0.10546875# disk.nvme0c0n1.in‚ 2669.08984375! disk.nvme0n1.in‚ 2669.08984375 disk.rbd0.out‚0 disk.dm-1.in‚ 0.10546875* disk./.usagePercent‚39.614428691128914 disk.rbd1.out‚4.6875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.19250011444092 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda3.out‚46.53125 disk.drbd2165.in‚0$ disk.nvme0n1.out‚4278.3232421875 network.recv‚641936 disk.dm-1.out‚46.53125! disk.dm-0.out‚4278.3232421875 disk.sda2.in‚0 disk.dm-0.in‚ 2669.08984375( proc.memory.availableMB‚ 132.30078125 network.sent‚ 5.316453e+06 disk.sda.out‚46.53125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.10546875Ðð÷2œ- _wandbruntime‚1948  _runtime‚1948™PovÅ¿ trainer/global_step‚4099 % _timestamp‚1.7843700070959082e+09 _runtime‚1948.539811846 _step‚329 % lr-AdamW/pg1‚0.000833371883680341 % lr-AdamW/pg2‚0.000833371883680341°zUHÊžÄ % lr-AdamW/pg1‚0.000833371883680341 % lr-AdamW/pg2‚0.000833371883680341 trainer/global_step‚4099 % _timestamp‚1.7843700070959082e+09 _runtime‚1948.539811846 _step‚329ɤri½2Ÿ- _wandbruntime‚1948  _runtime‚1948¥' PÊ ÈO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156968556875776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150225392863232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150225183266816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843700070970502e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_poolAJs%.allocated‚30782689 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156968556875776 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38928433 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30782687 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18908745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154330543982592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2411359442566 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69701143 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18908748 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28236732 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159388445045248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419871130112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38918456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147725309321216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500083542016 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9328213 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156968573915136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2499898849280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156741920464518 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69711122 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚1948.541018394 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156968573915136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159388340068864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30782689 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159388340068864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154330561021952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147725284417536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419871130112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2419783193088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4099 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28236961 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9327987 _step‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2419783193088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159388445045248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296xžiåÓ'¡ÍO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159388445045248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156968556875776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150225392863232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150225183266816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843700070970502e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30782689 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156968556875776 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38928433 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30782687 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18908745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154330543982592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2411359442566 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69701143 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18908748 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28236732 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159388445045248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419871130112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38918456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147725309321216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500083542016 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9328213 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156968573915136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2499898849280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156741920464518 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69711122 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚1948.541018394 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156968573915136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159388340068864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30782689 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159388340068864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154330561021952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147725284417536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419871130112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2419783193088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4099 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28236961 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9327987 _step‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2419783193088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144Êàä”±2¢- _wandbruntime‚1948  _runtime‚1948µL,×&£ÑM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147760220835840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38937728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9330385 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147760195932160 $ _timestamp‚1.784370007437845e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157005667840000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500488140800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69727638 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4099 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2411926764208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38927737 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150260684072960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159426020392448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28243617 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154367033057280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69717645 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159426125376000 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28243382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500660243456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9330153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2420440496640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420352552448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38937728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 _step‚331 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30789910 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30789908 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚156778856644926 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚154367016017920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69727638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18913229 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150260881079296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2411840627006 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚156778959821488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _runtime‚1948.881910436 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2420352552448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2420440496640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0|ñÔý ¤ÖM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚156778856644926 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚154367016017920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69727638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18913229 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150260881079296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2411840627006 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚156778959821488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _runtime‚1948.881910436 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2420352552448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2420440496640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStn} gßatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147760220835840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38937728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9330385 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147760195932160 $ _timestamp‚1.784370007437845e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157005667840000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500488140800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69727638 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4099 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2411926764208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38927737 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150260684072960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159426020392448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28243617 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154367033057280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69717645 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159426125376000 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28243382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500660243456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9330153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2420440496640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420352552448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38937728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 _step‚331 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30789910 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30789908 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346Ë%þ«2¥- _wandbruntime‚1948  _runtime‚1948Eì⦠2loss/FBt2/assignment_loss‚0.005761082484208184 .loss/FRt1/detection_loss‚0.606092450998041 /loss/FBt2/detection_loss‚0.4764924875318842 %loss/total_loss‚9.292916434496096 0loss/FRt2/assignment_loss‚0.7810523918696812 2loss/SRqqt1/assignment_loss‚0.7639774664232032 -loss/FRt2/detection_loss‚1.50464696254778 /loss/FBt1/detection_loss‚0.9889463449899967 epoch‚5 _step‚332 0loss/FRt1/assignment_loss‚1.8690520010003642 3loss/SRqqt2/assignment_loss‚0.49059762542373203 1loss/SRqqt1/detection_loss‚1.0366653389308924 2loss/FBt1/assignment_loss‚0.013535510434429363 1loss/SRqqt2/detection_loss‚0.7560967718618833 trainer/global_step‚4099 % _timestamp‚1.7843700074395177e+09 _runtime‚1948.883059306ðs‰3 §† -loss/FRt2/detection_loss‚1.50464696254778 /loss/FBt1/detection_loss‚0.9889463449899967 epoch‚5 _step‚332 0loss/FRt1/assignment_loss‚1.8690520010003642 3loss/SRqqt2/assignment_loss‚0.49059762542373203 1loss/SRqqt1/detection_loss‚1.0366653389308924 2loss/FBt1/assignment_loss‚0.013535510434429363 1loss/SRqqt2/detection_loss‚0.7560967718618833 trainer/global_step‚4099 % _timestamp‚1.7843700074395177e+09 _runtime‚1948.883059306 2loss/FBt2/assignment_loss‚0.005761082484208184 .loss/FRt1/detection_loss‚0.606092450998041 /loss/FBt2/detection_loss‚0.4764924875318842 %loss/total_loss‚9.292916434496096 0loss/FRt2/assignment_loss‚0.7810523918696812 2loss/SRqqt1/assignment_loss‚0.7639774664232032ÌH§‘¨‚(:† ئíÒ¶Ïአgpu.0.gpu‚49.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;K1–É©‚(:¾ ئíÒÏ” network.sent‚ 5.37902e+06* disk./.usagePercent‚39.614499605215904 disk.sda.out‚46.78125 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚4281.0185546875 disk.rbd0.out‚0 disk.dm-1.out‚46.78125 proc.memory.rssMB‚ 2695.1875$ disk./.usageGB‚590.2846717834473 disk.sda.in‚ 0.10546875 disk.rbd1.out‚4.75! disk.nvme0n1.in‚ 2669.08984375$ disk.nvme0n1.out‚4281.0185546875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚26.97518772209823# disk.nvme0c0n1.in‚ 2669.08984375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875! disk.dm-0.out‚4281.0185546875* proc.memory.percent‚16.450119018554688 network.recv‚646929 disk.sda3.in‚ 0.10546875 disk.sda3.out‚46.78125 disk.drbd2165.out‚0 disk.dm-0.in‚ 2669.08984375# memory_percent‚99.1922378540039% proc.memory.availableMB‚ 132.34375 proc.cpu.threads‚17¯HA ª‚(:„ ç¦íÒ¹ ÂŠ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚138.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d‘ù”Û«‚(:Ð ç¦íÒ³’$ memory_percent‚99.19264316558838$ disk./.usageGB‚590.2848739624023' disk.nvme0c0n1.out‚4285.74072265625 disk.dm-0.in‚ 2669.08984375 disk.sda3.out‚ 47.19921875* proc.memory.percent‚16.450119018554688 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.out‚ 47.19921875 disk.sda1.in‚0 network.recv‚652347 disk.dm-1.in‚ 0.10546875 disk.sda.in‚ 0.10546875 disk.drbd2165.out‚0( proc.memory.availableMB‚ 132.27734375 disk.rbd1.in‚0! disk.nvme0n1.in‚ 2669.08984375% disk.nvme0n1.out‚4285.74072265625 disk.rbd0.out‚0 disk.sda2.out‚0" disk.dm-0.out‚4285.74072265625* disk./.usagePercent‚39.614513173615215 disk.rbd1.out‚4.75 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2695.1875 cpu‚26.969660163693604 network.sent‚ 5.441608e+06 disk.sda.out‚ 47.19921875# disk.nvme0c0n1.in‚ 2669.08984375 disk.sda1.out‚0 disk.sda2.in‚0¶èrF2¬- _wandbruntime‚1970  _runtime‚1970n€ñÇ­Á _step‚333 & lr-AdamW/pg1‚0.0008329258099499663 & lr-AdamW/pg2‚0.0008329258099499663 trainer/global_step‚4149 % _timestamp‚1.7843700290089037e+09 _runtime‚1970.453628512Òºl]Ì®Æ _step‚333 & lr-AdamW/pg1‚0.0008329258099499663 & lr-AdamW/pg2‚0.0008329258099499663 trainer/global_step‚4149 % _timestamp‚1.7843700290089037e+09 _runtime‚1970.453628512ÍŸë 2¯- _wandbruntime‚1970  _runtime‚1970´ž**Î'°ÈO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158824122126336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9436756 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31143737 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19132948 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2529570434048 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39384358 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156154145748992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161272493481472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161272388505088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149470885052416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _runtime‚1970.453762745 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70538074 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158593817034176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39394335 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158824105086976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39394335 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31143739 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529402518528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70528095 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2448371355136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31143737 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2439671285184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19132945 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31143739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9437005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28569701 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161272388505088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149470885052416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _runtime‚1970.453762745 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70538074 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158593817034176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39394335 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158824105086976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39394335 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31143739 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529402518528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70528095 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2448371355136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31143737 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2439671285184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19132945 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31143739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9437005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28569701 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28576427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚d+¸Uú$9438998 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4149 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156190634823680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149505771663360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161310173812224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158861216051200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70554590 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31150960 % _timestamp‚1.7843700293463714e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70544597 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156190617784320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158861233090560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28576658 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161310173812224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158630959567596 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158630856391034 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161310068828672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39403630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152035761290240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2529989626880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39403630 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158861233090560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2440324743916 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31150960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2448852777472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2448852777472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2530163826688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2448940721664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19137432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39393639 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152035960393728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1970.790473484 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2440238606714 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0q²±@Ý&´×M _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149505771663360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161310173812224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158861216051200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70554590 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31150960 % _timestamp‚1.7843700293463714e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70544597 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156190617784320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158861233090560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28576658 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161310173812224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158630959567596 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158630856391034 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161310068828672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39403630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152035761290240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2529989626880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39403630 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158861233090560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2440324743916 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31150960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2448852777472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2448852777472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2530163826688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2448940721664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19137432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39393639 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152035960393728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1970.790473484 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2440238606714 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2448940721664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28576427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9438998 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4149 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156190634823680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327ÏÆvrð2µ- _wandbruntime‚1970  _runtime‚1970¨Cç4¶‚ 2loss/SRqqt1/assignment_loss‚0.7864359338684924 /loss/FBt1/detection_loss‚0.8369069374033383 epoch‚5 % _timestamp‚1.7843700293480988e+09 _runtime‚1970.791641137 _step‚336 0loss/FRt2/assignment_loss‚0.8721073114491523 1loss/SRqqt2/assignment_loss‚0.740400953644841 1loss/SRqqt1/detection_loss‚0.9338690373482126 trainer/global_step‚4149 /loss/FRt1/assignment_loss‚1.904713215189986 3loss/FBt2/assignment_loss‚0.0009975463808002817 1loss/SRqqt2/detection_loss‚0.7225815637367535 3loss/FBt1/assignment_loss‚0.0004690144173602156 .loss/FRt1/detection_loss‚0.577072043320328 /loss/FRt2/detection_loss‚1.6523467315450073 /loss/FBt2/detection_loss‚0.5485071282495152 %loss/total_loss‚9.576407416553788M ³… ·‡ .loss/FRt1/detection_loss‚0.577072043320328 /loss/FRt2/detection_loss‚1.6523467315450073 /loss/FBt2/detection_loss‚0.5485071282495152 %loss/total_loss‚9.576407416553788 2loss/SRqqt1/assignment_loss‚0.7864359338684924 /loss/FBt1/detection_loss‚0.8369069374033383 epoch‚5 % _timestamp‚1.7843700293480988e+09 _runtime‚1970.791641137 _step‚336 0loss/FRt2/assignment_loss‚0.8721073114491523 1loss/SRqqt2/assignment_loss‚0.740400953644841 1loss/SRqqt1/detection_loss‚0.9338690373482126 trainer/global_step‚4149 /loss/FRt1/assignment_loss‚1.904713215189986 3loss/FBt2/assignment_loss‚0.0009975463808002817 1loss/SRqqt2/detection_loss‚0.7225815637367535 3loss/FBt1/assignment_loss‚0.0004690144173602156Бê+¸‚(:… ö¦íÒºŒ gpu.0.gpu‚61.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚183.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.611 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓKÇ?×¹‚(:Ì ö¦íÒéêÅ disk.rbd1.out‚4.8125 disk.sda2.in‚0" disk.dm-0.out‚4298.26513671875 disk.sda.in‚ 0.10546875 proc.cpu.threads‚17 network.sent‚ 5.446546e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 2669.08984375 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 2669.08984375 disk.sda2.out‚0 disk.dm-1.out‚ 49.2265625 disk.sda.out‚ 49.2265625$ disk./.usageGB‚590.2885856628418 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 49.2265625 network.recv‚655498) disk./.usagePercent‚39.61476226894603 disk.drbd2165.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.450119018554688% disk.nvme0n1.out‚4298.26513671875' proc.memory.availableMB‚ 132.2578125 proc.memory.rssMB‚ 2695.1875# disk.nvme0c0n1.in‚ 2669.08984375 disk.sda1.out‚0 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 cpu‚26.96414729030361' disk.nvme0c0n1.out‚4298.26513671875$ memory_percent‚99.19276237487793½ìs"2º- _wandbruntime‚1992  _runtime‚1992|œ°iÅ»¿ % _timestamp‚1.7843700507077222e+09 _runtime‚1992.151634708 _step‚337 % lr-AdamW/pg1‚0.000832473186928991 % lr-AdamW/pg2‚0.000832473186928991 trainer/global_step‚4199>ƒì°Ê¼Ä % lr-AdamW/pg1‚0.000832473186928991 % lr-AdamW/pg2‚0.000832473186928991 trainer/global_step‚4199 % _timestamp‚1.7843700507077222e+09 _runtime‚1992.151634708 _step‚337ÑF-82½- _wandbruntime‚1992  _runtime‚1992|½24&¾ÇO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157977764554752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2476871580160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71365026 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31504789 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163156541917696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19357145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2558763353088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157977747515392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468069264892 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚338 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71355047 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163156436941312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163156436941312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31504787 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19357148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9545966 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2558943851520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160445919956734 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39860237 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31504787 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39850260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163156541917696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31504789 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160445816780284 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39860237 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151216435879936 _runtime‚1992.152871463 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160679670337536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153775404635136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160679670337536 WKDeviceStatsMonitor.on_train_batch_starÄ©Z™t/inactive_split.small_pool.allocated‚9546208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 $ _timestamp‚1.784370050708885e+09¹¯ªŠÒ'¿ÌO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2476871580160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71365026 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31504789 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163156541917696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19357145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2558763353088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157977747515392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468069264892 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚338 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71355047 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163156436941312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163156436941312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31504787 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19357148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9545966 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2558943851520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160445919956734 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39860237 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31504787 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39850260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163156541917696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31504789 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160445816780284 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39860237 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151216435879936 _runtime‚1992.152871463 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160679670337536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153775404635136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160679670337536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9546208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 $ _timestamp‚1.784370050708885e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157977764554752Ò5Àˆ;2À- _wandbruntime‚1992  _runtime‚1992â PRØ&ÁÒM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9548045 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163194222248448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151251372298240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚163194222248448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160716781301760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160716764262400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2468722723624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71371549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153810901721088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39869532 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31512010 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 % _timestamp‚1.7843700510506284e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19361632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚1992.494701293 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39859541 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28909674 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2477440946688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160716764262400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2477353002496 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163194117264896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2559348931584 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2559529422848 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31512010 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71381542 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160482856137142 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158014219550720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160716781301760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28909901 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153810696326144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160482959313704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160716764262400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2477353002496 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163194117264896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2559348931584 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2559529422848 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31512010 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71381542 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160482856137142 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158014219550720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160716781301760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28909901 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153810696326144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160482959313704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚163194222248448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160716781301760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160716764262400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2468722723624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71371549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153810901721088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39869532 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31512010 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 % _timestamp‚1.7843700510506284e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19361632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚1992.494701293 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39859541 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28909674 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2477440946688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ÓA·q2Ã- _wandbruntime‚1992  _runtime‚1992 ; |Ä€ _runtime‚1992.495890678 /loss/FRt2/assignment_loss‚0.845279676494775 2loss/FBt2/assignment_loss‚0.037222120734231826 % _timestamp‚1.7843700510523539e+09 0loss/FRt1/assignment_loss‚1.9131686166374984 /loss/FRt2/detection_loss‚1.5543154123308207 trainer/global_step‚4199 1loss/SRqqt1/detection_loss‚0.8742388082225678 /loss/FBt1/detection_loss‚0.5147805051191857 %loss/total_loss‚9.142209751267922 _step‚340 2loss/SRqqt1/assignment_loss‚0.7182689336654653 2loss/SRqqt2/assignment_loss‚0.5553163552497246 1loss/FBt1/assignment_loss‚0.11916463236967072 /loss/FRt1/detection_loss‚0.6264361636795637 1loss/SRqqt2/detection_loss‚0.7927838612426668 .loss/FBt2/detection_loss‚0.591234665521751 epoch‚5¿dâ Å… 1loss/SRqqt1/detection_loss‚0.8742388082225678 /loss/FBt1/detection_loss‚0.5147805051191857 %loss/total_loss‚9.142209751267922 _step‚340 2loss/SRqqt1/assignment_loss‚0.7182689336654653 2loss/SRqqt2/assignment_loss‚0.5553163552497246 1loss/FBt1/assignment_loss‚0.11916463236967072 /loss/FRt1/detection_loss‚0.6264361636795637 1loss/SRqqt2/detection_loss‚0.7927838612426668 .loss/FBt2/detection_loss‚0.591234665521751 epoch‚5 _runtime‚1992.495890678 /loss/FRt2/assignment_loss‚0.845279676494775 2loss/FBt2/assignment_loss‚0.037222120734231826 % _timestamp‚1.7843700510523539e+09 0loss/FRt1/assignment_loss‚1.9131686166374984 /loss/FRt2/detection_loss‚1.5543154123308207 trainer/global_step‚4199Ôß:j&‘Æ‚(:† …§íң粊 gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚183.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ôý”Ç‚(:Î …§íÒ«˜¼Ž network.sent‚ 5.522214e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 49.56640625 cpu‚26.959073248317473 proc.cpu.ŒÀEthreads‚17* disk./.usagePercent‚39.614806558249455 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0" disk.dm-0.out‚4307.27685546875$ disk./.usageGB‚590.2892456054688 disk.drbd2165.out‚0 disk.sda.in‚ 0.10546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.10546875 disk.rbd1.out‚4.8125 disk.nvme0n1.in‚ 2781.8046875 disk.sda1.in‚0' disk.nvme0c0n1.out‚4307.27685546875 disk.sda.out‚ 49.56640625% disk.nvme0n1.out‚4307.27685546875 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2781.8046875$ memory_percent‚99.19266700744629 proc.memory.rssMB‚ 2695.1875* proc.memory.percent‚16.450119018554688 network.recv‚660692 disk.sda3.out‚ 49.56640625" disk.nvme0c0n1.in‚ 2781.8046875 disk.sda2.in‚0' proc.memory.availableMB‚ 132.2734375µ„ÍÇÈ‚(:… ”§íÒƛŊ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚179.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h…Y°É‚(:¥ ”§íÒí¯ˆ" disk.nvme0c0n1.in‚ 2781.8046875! disk.nvme0n1.out‚ 4311.6953125 disk.sda2.out‚0) disk./.usagePercent‚39.61479964604603 disk.drbd2165.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 2781.8046875 cpu‚26.953646491311435 disk.drbd2165.out‚0 disk.rbd1.out‚4.875 network.recv‚665495$ disk./.usageGB‚590.2891426086426 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.450119018554688 disk.rbd1.in‚0# disk.nvme0c0n1.out‚ 4311.6953125 disk.sda1.in‚0 disk.dm-1.out‚50$ memory_percent‚99.19314384460449 network.sent‚ 5.572028e+06 disk.nvme0n1.in‚ 2781.8046875 disk.dm-0.out‚ 4311.6953125 disk.sda3.out‚50' proc.memory.availableMB‚ 132.1953125 proc.memory.rssMB‚ 2695.1875 disk.dm-1.in‚ 0.10546875 disk.sda3.in‚ 0.10546875 disk.sda.in‚ 0.10546875 disk.sda.out‚50N¡/2Ê- _wandbruntime‚2014  _runtime‚2014¬ÍgäÇËÁ & lr-AdamW/pg1‚0.0008320140218670915 & lr-AdamW/pg2‚0.0008320140218670915 trainer/global_step‚4249 % _timestamp‚1.7843700727608595e+09 _runtime‚2014.204635439 _step‚341ÈzÌÌÆ % _timestamp‚1.7843700727608595e+09 _runtime‚2014.204635439 _step‚341 & lr-AdamW/pg1‚0.0008320140218670915 & lr-AdamW/pg2‚0.0008320140218670915 trainer/global_step‚4249Õµ`E2Í- _wandbruntime‚2014  _runtime‚2014èÂÎ'ÎÈO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19581348 _step‚342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2505283868160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2014.205866734 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29236780 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496553381690 trainer/global_step‚4249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165040590353920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162535201509376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40326139 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72191978 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29237015 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162535218548736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2588368794112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152962036514816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31865837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843700727618775e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2505371805184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165040485377536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31865839 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159801349281792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165040590353920 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72181999 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9655435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162297919702842 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2496467244600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2014.205866734 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29236780 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496553381690 trainer/global_step‚4249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165040590353920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162535201509376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40326139 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72191978 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29237015 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162535218548736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2588368794112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152962036514816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31865837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843700727618775e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2505371805184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165040485377536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31865839 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159801349281792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165040590353920 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72181999 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9655435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162297919702842 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2496467244600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29243429 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19585832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2497034566130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72208494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19585829 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2588774877696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72198501 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31873060 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497120703332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40335434 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162572312473600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2505941171712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165078270684672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9657830 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2588955368960 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2505853227520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165078165701120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162572312473600 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155585903398400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159837821317120 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31873060 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40325443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505853227520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165078270684672 GADeviceStatsMonitor.on_train_batch_en (sd/oversize_segments.allocated‚0 _step‚343 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72198501 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152996948029440r,ƒÜ&ÒÖM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31873058 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29243429 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19585832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2497034566130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72208494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19585829 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2588774877696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72198501 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31873060 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497120703332 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40335434 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162572312473600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2505941171712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165078270684672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9657830 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2588955368960 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2505853227520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165078165701120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162572312473600 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155585903398400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159837821317120 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31873060 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40325443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505853227520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165078270684672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚343 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72198501 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152996948029440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162334959059812×>?/¬2Ó- _wandbruntime‚2014  _runtime‚2014Rù/oÔý 0loss/FRt2/assignment_loss‚0.9381974765658379 0loss/FBt1/assignment_loss‚0.1559006341423128 .loss/FBt2/detection_loss‚0.604861859410044 2loss/SRqqt1/assignment_loss‚0.6064326205308764 %loss/total_loss‚9.415190170894912 epoch‚5 _runtime‚2014.549231038 0loss/FRt1/assignment_loss‚1.8522487695484968 2loss/SRqqt2/assignment_loss‚0.6500929640563193 0loss/SRqqt1/detection_loss‚0.808004482033161 trainer/global_step‚4249 _step‚344 2loss/FBt2/assignment_loss‚0.012934553447667203 .loss/FRt1/detection_loss‚0.552348403053839 /loss/FRt2/detection_loss‚1.4995123786013573 1loss/SRqqt2/detection_loss‚0.7705614068516833 /loss/FBt1/detection_loss‚0.9640946226533164 $ _timestamp‚1.784370073105692e+09u¿(Õ‚ /loss/FRt2/detection_loss‚1.4995123786013573 1loss/SRqqt2/detection_loss‚0.7705614068516833 /loss/FBt1/detection_loss‚0.9640946226533164 $ _timestamp‚1.784370073105692e+09 0loss/FRt2/assignment_loss‚0.9381974765658379 0loss/FBt1/assignment_loss‚0.1559006341423128 .loss/FBt2/detection_loss‚0.604861859410044 2loss/SRqqt1/assignment_loss‚0.6064326205308764 %loss/total_loss‚9.415190170894912 epoch‚5 _runtime‚2014.549231038 0loss/FRt1/assignment_loss‚1.8522487695484968 2loss/SRqqt2/assignment_loss‚0.6500929640563193 0loss/SRqqt1/detection_loss‚0.808004482033161 trainer/global_step‚4249 _step‚344 2loss/FBt2/assignment_loss‚0.012934553447667203 .loss/FRt1/detection_loss‚0.552348403053839Øe¥ ‘Ö‚(:† £§íÒ²£ãŠ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚170.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂPóeÕׂ(:Ê £§íÒ¬š¨Œ disk.drbd2165.out‚0% disk.nvme0n1.out‚4318.23583984375 disk.sda1.out‚0' proc.memory.availableMB‚ 131.9453125 proc.memory.rssMB‚ 2695.1875 cpu‚26.9485251833801$ disk./.usageGB‚590.2903861999512 disk.rbd1.out‚4.875 disk.rbd0.in‚0$ memory_percent‚99.19466972351074 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 50.20703125 disk.nvme0n1.in‚ 2781.8046875 disk.dm-1.in‚ 0.10546875 disk.sda.out‚ 50.20703125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2781.8046875* proc.memory.percent‚16.450119018554688) disk./.usagePercent‚39.61488310450219' disk.nvme0c0n1.out‚4318.23583984375 proc.cpu.threads‚17 network.sent‚ 5.576981e+06 network.recv‚669284 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 2781.8046875 disk.sda1.in‚0 disk.sda.in‚ 0.10546875 disk.dm-1.out‚ 50.20703125" disk.dm-0.out‚4318.23583984375 disk.sda3.in‚ 0.10546875{MÃ2Ø- _wandbruntime‚2035  _runtime‚2035æ©–ÆÙÀ _runtime‚2035.999119532 _step‚345 & lr-AdamW/pg1‚0.0008315483221187282 & lr-AdamW/pg2‚0.0008315483221187282 trainer/global_step‚4299 $ _timestamp‚1.784370094555271e+09oÉbÕËÚÅ trainer/global_step‚4299 $ _timestamp‚1.784370094555271e+09 _runtime‚2035.999119532 _step‚345 & lr-AdamW/pg1‚0.0008315483221187282 & lr-AdamW/pg2‚0.0008315483221187282Ù&úÿ2Û- _wandbruntime‚2036  _runtime‚2036:§æÔÎ'ÜÈO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40782064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154707612246016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73018930 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2617838266880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19805548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2617664059904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73008951 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164390766759936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32226887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29570368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164390749720576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2533872030208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40782064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32226889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166924533813760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9764820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154707587342336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚2036.000326761 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32226887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157325450512896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 trainer/global_step‚4299 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2524951361398 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166924638790144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166924638790144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2533784093184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40792041 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73008951 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164390766759936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32226887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29570368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164390749720576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2533872030208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40782064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32226889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166924533813760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9764820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154707587342336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚2036.000326761 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32226887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157325450512896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 trainer/global_step‚4299 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2524951361398 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166924638790144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166924638790144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2533784093184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40792041 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2617838266880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19805548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2617664059904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166962319120896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40791345 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154742498856960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29576815 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164427860684800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9767025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2618248362496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32234110 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2525518683040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _step‚347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19810029 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2534353452544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2618428853760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73035446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164427877724160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157360952614400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161661440122880 % _timestamp‚1.7843700948944283e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164427877724160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161661423083520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚3276ŽËÝ&à×M >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _step‚347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19810029 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2534353452544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2618428853760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73035446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164427877724160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157360952614400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161661440122880 % _timestamp‚1.7843700948944283e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164427877724160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161661423083520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166962319120896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40791345 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154742498856960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29576815 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164427860684800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9767025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2618248362496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32234110 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2525518683040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144ÛÀqmé2á- _wandbruntime‚2036  _runtime‚2036v¤é†/loss/FBt1/detection_loss‚1.1147580501239966 %loss/total_loss‚9.469199247471073 % _timestamp‚1.7843700948960683e+09 _runtime‚2036.339608343 0loss/FRt2/assignment_loss‚0.8267934181751349 2loss/SRqqt2/assignment_loss‚0.5333226495960268 1loss/SRqqt2/detection_loss‚0.6641599286777904 epoch‚5 _step‚348 .loss/FRt1/assignment_loss‚2.00070380597417 2loss/FBt1/assignment_loss‚0.026245195385199087 1loss/FBt2/assignment_loss‚0.01895235387622135 1loss/SRqqt1/detection_loss‚0.9061585800224092 /loss/FBt2/detection_loss‚0.5119133814816561 trainer/global_step‚4299 /loss/FRt2/detection_loss‚1.6709128291465534 2loss/SRqqt1/assignment_loss‚0.5967315617234035 /loss/FRt1/detection_loss‚0.5985474932885113»aà™ ã… _step‚348 .loss/FRt1/assignment_loss‚2.00070380597417 2loss/FBt1/assignment_loss‚0.026245195385199087 1loss/FBt2/assignment_loss‚0.01895235387622135 1loss/SRqqt1/detection_loss‚0.9061585800224092 /loss/FBt2/detection_loss‚0.5119133814816561 trainer/global_step‚4299 /loss/FRt2/detection_loss‚1.6709128291465534 2loss/SRqqt1/assignment_loss‚0.5967315617234035 /loss/FRt1/detection_loss‚0.5985474932885113 /loss/FBt1/detection_loss‚1.1147580501239966 %loss/total_loss‚9.469199247471073 % _timestamp‚1.7843700948960683e+09 _runtime‚2036.339608343 0loss/FRt2/assignment_loss‚0.8267934181751349 2loss/SRqqt2/assignment_loss‚0.5333226495960268 1loss/SRqqt2/detection_loss‚0.6641599286777904 epoch‚5Üÿ§ÿ‘ä‚(:† ²§íÒŽÍ”Œ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚186.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zóÑå‚(:Æ ²§íÒÂÔó disk.sda.in‚ 0.10546875 disk.sda.out‚ 50.5078125 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚5 disk.rbd0.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.450119018554688 disk.sda3.in‚ 0.10546875 disk.sda3.out‚ 50.5078125" disk.nvme0c0n1.in‚ 2781.8046875 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 2781.8046875" disk.dm-0.out‚4322.93701171875 cpu‚26.943453989007185 network.sent‚ 5.639698e+06$ disk./.usageGB‚590.2902565002441 disk.rbd0.out‚0 disk.sda2.in‚0 network.recv‚674279 disk.nvme0n1.in‚ 2781.8046875 disk.sda2.out‚0 proc.memory.rssMB‚ 2695.1875 proc.cpu.threads‚17 disk.drbd2165.in‚0% disk.nvme0n1.out‚4322.93701171875 disk.dm-1.out‚ 50.5078125) disk./.usagePercent‚39.61487440024603' disk.nvme0c0n1.out‚4322.93701171875 disk.sda1.out‚0$ memory_percent‚99.19459819793701( proc.memory.availableMB‚ 131.95703125–ÿœæ‚(:‘ Á§íÒÿ›§Š gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚176.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.376999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÜéÎç‚(:à Á§íÒõŠôŒ disk.nvme0n1.in‚ 2849.3828125$ memory_percent‚99.19424057006836& proc.memory.availableMB‚ 132.015625 disk.sda.in‚ 0.10546875 disk.rbd0.in‚0 disk.sda1.out‚0" disk.dm-0.out‚4329.74267578125 proc.memory.rssMB‚ 2695.1875 network.sent‚ 5.70225e+06$ disk./.usageGB‚590.2904891967773 disk.sda2.out‚0' disk.nvme0c0n1.out‚4329.74267578125* disk./.usagePercent‚39.614890016705615 disk.sda3.in‚ 0.10546875 disk.sda.out‚ 50.7109375 disk.drbd2165.out‚0 disk.rbd1.out‚5 disk.rbd0.out‚0 disk.dm-1.out‚ 50.7109375 network.recv‚679523% disk.nvme0n1.out‚4329.74267578125 disk.sda2.in‚0 disk.dm-0.in‚ 2849.3828125 disk.sda3.out‚ 50.7109375 cpu‚26.93844772382213 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 2849.3828125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.10546875* proc.memory.percent‚16.450119018554688 disk.drbd2165.in‚0]šÕu2è- _wandbruntime‚2057  _runtime‚2057<ÙÓŒÆéÀ & lr-AdamW/pg1‚0.0008310760951430278 & lr-AdamW/pg2‚0.0008310760951430278 trainer/global_step‚4349 $ _timestamp‚1.784370116469933e+09 _runtime‚2057.913748699 _step‚349)àÇ ËêÅ & lr-AdamW/pg1‚0.0008310760951430278 & lr-AdamW/pg2‚0.0008310760951430278 trainer/global_step‚4349 $ _timestamp‚1.784370116469933e+09 _runtime‚2057.913748699 _step‚349Ý)íL?2ë- _wandbruntime‚2057  _runtime‚2057N±4 ÂìÇO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159100259133952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168808582249984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2553349341106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2553263204016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166246297931776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9873400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73845882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2562372255232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2562284318208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2057.918698202 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168808687226368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168808582249984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2562372255232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 trainer/global_step‚4349 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166246314971136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159100464536064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41257943 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156453187977216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20029745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166001919195058 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32587937 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32587937¶Ò'íÌO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41257943 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156453187977216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20029745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166001919195058 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32587937 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32587937 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159100259133952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168808582249984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2553349341106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2553263204016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166246297931776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9873400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73845882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2562372255232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2562284318208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2057.918698202 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168808687226368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168808582249984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2562372255232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 trainer/global_step‚4349 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166246314971136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159100464536064ÞÇ 2î- _wandbruntime‚2058  _runtime‚2058M´¶¹Ø&ïÒM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20034229 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2553830525546 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29909535 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2647687012864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166038958552028 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2553916662748 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156488099491840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73862398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41267238 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166283408896000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32595160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2562941621760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41257247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168846367557120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32595158 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159135761601024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163485041889280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚2058.259010363 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73862398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156488074588160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168846367557120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2562853677568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32595160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166283408896000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2562941621760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159135962801664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2562853677568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32595158 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166283425935360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2647863309824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 % _timestamp‚1.7843701168148725e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991C?}ö!ð×M EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156488099491840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73862398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41267238 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166283408896000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32595160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2562941621760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41257247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168846367557120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32595158 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159135761601024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163485041889280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚2058.259010363 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73862398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156488074588160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168846367557120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2562853677568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32595160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166283408896000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2562941621760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159135962801664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2562853677568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32595158 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166283425935360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2647863309824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 % _timestamp‚1.7843701168148725e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.curre\¢¶rçnt‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20034229 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2553830525546 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29909535 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2647687012864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166038958552028 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2553916662748 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ßiõ/2ñ- _wandbruntime‚2058  _runtime‚2058ßàbãòü /loss/FBt1/detection_loss‚0.8652243897042893 trainer/global_step‚4349 _runtime‚2058.260398624 1loss/FBt1/assignment_loss‚0.05062045793663275 /loss/FRt2/detection_loss‚1.6465369683227002 %loss/total_loss‚9.372308358709173 _step‚352 .loss/FBt2/detection_loss‚0.511117475843057 2loss/SRqqt1/assignment_loss‚0.7003968911999205 1loss/FBt2/assignment_loss‚0.08513972534389191 1loss/SRqqt2/detection_loss‚0.6834459470954003 .loss/FRt1/detection_loss‚0.597816074780177 epoch‚5 # _timestamp‚1.78437011681686e+09 /loss/FRt1/assignment_loss‚1.892137691466076 0loss/FRt2/assignment_loss‚0.9181754920870212 2loss/SRqqt2/assignment_loss‚0.6049324109666508 1loss/SRqqt1/detection_loss‚0.8167648339633551+×».ó %loss/total_loss‚9.372308358709173 _step‚352 .loss/FBt2/detection_loss‚0.511117475843057 2loss/SRqqt1/assignment_loss‚0.7003968911999205 1loss/FBt2/assignment_loss‚0.08513972534389191 1loss/SRqqt2/detection_loss‚0.6834459470954003 .loss/FRt1/detection_loss‚0.597816074780177 epoch‚5 # _timestamp‚1.78437011681686e+09 /loss/FRt1/assignment_loss‚1.892137691466076 0loss/FRt2/assignment_loss‚0.9181754920870212 2loss/SRqqt2/assignment_loss‚0.6049324109666508 1loss/SRqqt1/detection_loss‚0.8167648339633551 /loss/FBt1/detection_loss‚0.8652243897042893 trainer/global_step‚4349 _runtime‚2058.260398624 1loss/FBt1/assignment_loss‚0.05062045793663275 /loss/FRt2/detection_loss‚1.6465369683227002àzÊye‘ô‚(:† ЧíÒŽæö‹ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚175.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç˜=×õ‚(:Ì ЧíÒæ¹– disk.sda.in‚ 0.10546875 network.sent‚ 5.707234e+06 disk.sda3.out‚ 51.17578125 disk.rbd1.out‚5.0625 disk.rbd0.in‚0) disk./.usagePercent‚39.61488771263781" disk.dm-0.out‚4333.64697265625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 2849.3828125' disk.nvme0c0n1.out‚4333.64697265625 disk.rbd0.out‚0 disk.dm-0.in‚ 2849.3828125 disk.rbd1.in‚0* proc.memory.percent‚16.450119018554688' proc.memory.availableMB‚ 132.2421875 disk.sda.out‚ 51.17578125# disk./.usageGB‚590.290454864502 disk.sda2.out‚0 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚ 51.17578125$ memory_percent‚99.19285774230957 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.10546875 disk.drbd2165.out‚0% disk.nvme0n1.out‚4333.64697265625 cpu‚26.933398502236056 proc.cpu.threads‚17 network.recv‚682662 disk.nvme0n1.in‚ 2849.3828125 proc.memory.rssMB‚ 2695.1875IÀ ñ2ö- _wandbruntime‚2079  _runtime‚2079–­Æ÷À _runtime‚ 2079.98200549 _step‚353 & lr-AdamW/pg1‚0.0008305973485036638 & lr-AdamW/pg2‚0.0008305973485036638 trainer/global_step‚4399 % _timestamp‚1.7843701385381796e+09QT}àËøÅ _runtime‚ 2079.98200549 _step‚353 & lr-AdamW/pg1‚0.0008305973485036638 & lr-AdamW/pg2‚0.0008305973485036638 trainer/global_step‚4399 % _timestamp‚1.7843701385381796e+09á¬l€J2ù- _wandbruntime‚2079  _runtime‚2079`~|‰Î'úÈO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170692630686208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168101863182336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2676553071104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167853918941166 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158198763708416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160875291875840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165272171620352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170692735662592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2079.983373723 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32948987 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170692735662592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168101846142976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2590784543232 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170692630686208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167853815764716 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9982498 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9982729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2590784543232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41713868 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168101863182336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20253948 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2581747320814 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160875480500736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚354 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20253945 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0JpÃ8Ó'ûÍO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170692735662592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168101846142976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2590784543232 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170692630686208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167853815764716 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9982498 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9982729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2590784543232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41713868 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168101863182336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20253948 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2581747320814 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160875480500736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚354 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20253945 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170692630686208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168101863182336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2676553071104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167853918941166 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158198763708416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160875291875840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165272171620352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170692735662592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2079.983373723 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32948987â3Æß2ü- _wandbruntime‚2080  _runtime‚2080ø~ÊÇLýÒM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 % _timestamp‚1.7843701388874872e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170730415993344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32956210 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20258429 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9984936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170730415993344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168138957107200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74689350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2677135905280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165308643655680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32956208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168138974146560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30243123 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167890958298136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2591353902592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9984694 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74679357 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2582314642456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160910989522432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20258432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160910786224640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4399 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552ëç]¾Œ  [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168138974146560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158233650319360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2582228505254 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74689350 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2591353902592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41723149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F©ŒÝ&þ×M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20258429 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9984936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170730415993344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168138957107200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74689350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2677135905280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165308643655680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32956208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168138974146560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30243123 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167890958298136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2591353902592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9984694 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74679357 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2582314642456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160910989522432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20258432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160910786224640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4399 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168138974146560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158233650319360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2582228505254 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74689350 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2591353902592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41723149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192ãõD_•2ÿ- _wandbruntime‚2080  _runtime‚2080xZû€€ 0loss/FRt2/assignment_loss‚0.9056604879243033 2loss/FBt2/assignment_loss‚0.025022550799395523 /loss/FBt2/detection_loss‚0.6461449328581897 %loss/total_loss‚9.352018820131168 2loss/FBt1/assignment_loss‚0.024735506940227685 .loss/FRt2/detection_loss‚1.764243443645537 0loss/SRqqt2/detection_loss‚0.799965605395846 epoch‚5 trainer/global_step‚4399 _step‚356 0loss/FRt1/assignment_loss‚1.8552378434253125 2loss/SRqqt2/assignment_loss‚0.6276910144649689 /loss/FRt1/detection_loss‚0.5704451178559403 _runtime‚ 2080.33290691 2loss/SRqqt1/assignment_loss‚0.6271775668779889 1loss/SRqqt1/detection_loss‚0.8567642180613525 /loss/FBt1/detection_loss‚0.6489305318821044 % _timestamp‚1.7843701388889995e+09ójª … 0loss/FRt1/assignment_loss‚1.8552378434253125 2loss/SRqqt2/assignment_loss‚0.6276910144649689 /loss/FRt1/detection_loss‚0.5704451178559403 _runtime‚ 2080.33290691 2loss/SRqqt1/assignment_loss‚0.6271775668779889 1loss/SRqqt1/detection_loss‚0.8567642180613525 /loss/FBt1/detection_loss‚0.6489305318821044 % _timestamp‚1.7843701388889995e+09 0loss/FRt2/assignment_loss‚0.9056604879243033 2loss/FBt2/assignment_loss‚0.025022550799395523 /loss/FBt2/detection_loss‚0.6461449328581897 %loss/total_loss‚9.352018820131168 2loss/FBt1/assignment_loss‚0.024735506940227685 .loss/FRt2/detection_loss‚1.764243443645537 0loss/SRqqt2/detection_loss‚0.799965605395846 epoch‚5 trainer/global_step‚4399 _step‚356䬜ƒ+‘‚‚(:† ߧíÒôØŠ gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚179.923' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Rö¢Öƒ‚(:Ë ߧíÒ—í disk.rbd0.in‚0 network.recv‚687910$ disk./.usageGB‚590.2903442382812 disk.sda.in‚ 0.10546875 disk.sda.out‚ 51.25390625 disk.drbd2165.in‚0 disk.rbd0.out‚0 cpu‚26.9285282753072 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-1.out‚ 51.25390625 disk.rbd1.out‚5.0625 disk.sda2.out‚0 proc.memory.rssMB‚ 2695.1875 disk.sda1.out‚0) disk./.usagePercent‚39.61488028841932 disk.sda3.in‚ 0.10546875" disk.nvme0c0n1.in‚ 2849.3828125' disk.nvme0c0n1.out‚4339.33154296875 disk.nvme0n1.in‚ 2849.3828125* proc.memory.percent‚16.450119018554688" disk.dm-0.out‚4339.33154296875 disk.sda1.in‚0$ memory_percent‚99.19428825378418' proc.memory.availableMB‚ 132.0078125 disk.sda2.in‚0 network.sent‚ 5.769847e+06 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 2849.3828125 disk.sda3.out‚ 51.25390625 disk.drbd2165.out‚0% disk.nvme0n1.out‚4339.33154296875Ì1.P„‚(:… î§íÒë“–‹ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚175.085' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.017 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓyIÌ…‚(:Á î§íÒÆ‹Ž disk.sda1.out‚0 disk.dm-1.in‚ 0.10546875 network.recv‚692906" disk.nvme0c0n1.in‚ 2849.3828125% disk.nvme0c0n1.out‚4342.291015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.10546875# disk.nvme0n1.out‚4342.291015625 disk.sda.out‚ 51.578125 disk.drbd2165.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 132.20703125 cpu‚26.923856049833798 disk.sda3.out‚ 51.578125 disk.nvme0n1.in‚ 2849.3828125 disk.sda2.in‚0* proc.memory.percent‚16.450119018554688 proc.cpu.threads‚17 disk.dm-1.out‚ 51.578125) disk./.usagePercent‚39.61487875237411 disk.sda3.in‚ 0.10546875 disk.sda1.in‚0 disk.dm-0.in‚ 2849.3828125 disk.rbd1.in‚0$ memory_percent‚99.19307231903076 disk.rbd1.out‚5.125 network.sent‚ 5.832544e+06$ disk./.usageGB‚590.2903213500977 disk.dm-0.out‚4342.291015625 proc.memory.rssMB‚ 2695.1875 disk.drbd2165.in‚0Œu-2†- _wandbruntime‚2101  _runtime‚2101ÀñÛƇÀ _step‚357 & lr-AdamW/pg1‚0.0008301120898687355 & lr-AdamW/pg2‚0.0008301120898687355 trainer/global_step‚4449 $ _timestamp‚1.784370160326056e+09 _runtime‚2101.769859329êMÙèˈŠtrainer/global_step‚4449 $ _timestamp‚1.784370160326056e+09 _runtime‚2101.769859329 _step‚357 & lr-AdamW/pg1‚0.0008301120898687355 & lr-AdamW/pg2‚0.0008301120898687355åiÙ ©2‰- _wandbruntime‚2101  _runtime‚2101>ô(Ï'ŠÉO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2706222731776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159944314535936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20478145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172576679122432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2610145300522 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚358 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172576784098816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10091502 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 _runtime‚2101.771248822 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169957411393536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33310039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33310037 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2619284768256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 trainer/global_step‚4449 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42179770 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162650562171392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2706046427648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42189747 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20478148 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162650360963584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167095773386752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33310037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚169705918687274 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30569420 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30569650 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚169957411393536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2619284768256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2619372705280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75499786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10091275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159944339439616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2610059163432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0!•$ˆd‹ÎO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20478145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172576679122432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2610145300522 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚358 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172576784098816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10091502 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 _runtime‚2101.771248822 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169957411393536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33310039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33310037 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2619284768256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 trainer/global_step‚4449 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42179770 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚45;o¨'p2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162650562171392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2706046427648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42189747 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20478148 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162650360963584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167095773386752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33310037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚169705918687274 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30569420 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30569650 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚169957411393536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2619284768256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2619372705280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75499786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10091275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159944339439616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2610059163432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2706222731776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159944314535936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450æ…\«@2Œ- _wandbruntime‚2102  _runtime‚2102_M.áÚ&ÔM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚169994505318400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33317258 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42189051 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75506309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2619942071808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30576134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167132245422080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10093735 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172614359446016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75516302 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2706809490944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162686060445184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75506309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42199042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚169994522357760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2619854127616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159979250954240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169742958044244 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167132228382720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172614464429568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2619854127616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20482632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10093505 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159979226050560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚169994522357760 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2706633193984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33317260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33317258 % _timestamp‚1.7843701606715453e+09!Ùóß&ŽÙM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75506309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42199042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚169994522357760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2619854127616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159979250954240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169742958044244 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167132228382720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172614464429568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2619854127616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20482632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10093505 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159979226050560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚169994522357760 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2706633193984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33317260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33317258 % _timestamp‚1.7843701606715453e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚169994505318400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33317258 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42189051 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75506309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2619942071808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30576134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167132245422080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10093735 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172614359446016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75516302 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2706809490944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162686060445184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192çñ+2 2- _wandbruntime‚2102  _runtime‚2102&ÆŠþ 1loss/FBt1/assignment_loss‚0.03468553236594288 1loss/FBt2/assignment_loss‚0.04043809674613447 .loss/FBt2/detection_loss‚0.472510019052689 trainer/global_step‚4449 0loss/FRt1/assignment_loss‚1.8923889124348137 %loss/total_loss‚9.731385654437975 $ _timestamp‚1.784370160672873e+09 /loss/FRt1/detection_loss‚0.5900068034775583 0loss/SRqqt1/detection_loss‚0.962447421612671 epoch‚5 2loss/SRqqt1/assignment_loss‚0.7891749736437624 2loss/SRqqt2/assignment_loss‚0.6183337430747298 /loss/FRt2/detection_loss‚1.5614450935257274 1loss/SRqqt2/detection_loss‚0.7607791183325343 /loss/FBt1/detection_loss‚0.9639298399644238 _runtime‚2102.116472053 _step‚360 0loss/FRt2/assignment_loss‚1.0452461002069875¹Œ ‘ƒ %loss/total_loss‚9.731385654437975 $ _timestamp‚1.784370160672873e+09 /loss/FRt1/detection_loss‚0.5900068034775583 0loss/SRqqt1/detection_loss‚0.962447421612671 epoch‚5 2loss/SRqqt1/assignment_loss‚0.7891749736437624 2loss/SRqqt2/assignment_loss‚0.6183337430747298 /loss/FRt2/detection_loss‚1.5614450935257274 1loss/SRqqt2/detection_loss‚0.7607791183325343 /loss/FBt1/detection_loss‚0.9639298399644238 _runtime‚2102.116472053 _step‚360 0loss/FRt2/assignment_loss‚1.0452461002069875 1loss/FBt1/assignment_loss‚0.03468553236594288 1loss/FBt2/assignment_loss‚0.04043809674613447 .loss/FBt2/detection_loss‚0.472510019052689 trainer/global_step‚4449 0loss/FRt1/assignment_loss‚1.8923889124348137è¢E¯‘’‚(:† ý§íÒåÚЋ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)«˜Ú“‚(:Ï ý§íÒÙÿ° network.recv‚696409 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2695.1875 cpu‚26.919104441088898# disk./.usageGB‚590.290454864502 disk.rbd1.out‚5.125! disk.nvme0n1.in‚ 2905.93359375 disk.dm-1.in‚ 0.10546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.61488771263781 disk.dm-0.in‚ 2905.93359375" disk.dm-0.out‚4347.54443359375 disk.sda2.in‚0 disk.sda3.out‚ 51.80078125 disk.sda.out‚ 51.80078125' disk.nvme0c0n1.out‚4347.54443359375 disk.sda1.in‚0$ memory_percent‚99.19469356536865( proc.memory.availableMB‚ 131.94140625* proc.memory.percent‚16.450119018554688 network.sent‚ 5.850395e+06 disk.sda.in‚ 0.10546875# disk.nvme0c0n1.in‚ 2905.93359375% disk.nvme0n1.out‚4347.54443359375 disk.dm-1.out‚ 51.80078125 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.10546875 disk.rbd1.in‚0É…a½2”- _wandbruntime‚2123  _runtime‚2123d\˜ÜÇ•Á _step‚361 & lr-AdamW/pg1‚0.0008296203270106448 & lr-AdamW/pg2‚0.0008296203270106448 trainer/global_step‚4499 % _timestamp‚1.7843701823660207e+09 _runtime‚2123.809864242˜°®ðÌ–Æ % _timestamp‚1.7843701823660207e+09 _runtime‚2123.809864242 _step‚361 & lr-AdamW/pg1‚0.0008296203270106448 & lr-AdamW/pg2‚0.0008296203270106448 trainer/global_step‚4499é½òø÷2—- _wandbruntime‚2123  _runtime‚2123ö½5›¢˜ÉO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76326738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2735699832320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2647784993280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33671089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161689915170816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161689890267136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33671087 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174460832535040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174460727558656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33671089 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171812942565376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42645672 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _step‚362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10200307 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2647872930304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2638457143140 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2647784993280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171812959604736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42655649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171557815256932 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.oÍP-n_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174460832535040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2647872930304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168919358113792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 $ _timestamp‚1.784370182367139e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42655649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10200078 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174460727558656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171557918433382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638543280230 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20702348 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144úÙÔ'™ÎO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174460832535040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174460727558656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33671089 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171812942565376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42645672 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _step‚362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10200307 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2647872930304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2638457143140 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2647784993280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171812959604736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42655649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171557815256932 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174460832535040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2647872930304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168919358113792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 $ _timestamp‚1.784370182367139e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42655649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10200078 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174460727558656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171557918433382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638543280230 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20702348 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76326738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2735699832320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2647784993280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33671089 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161689915170816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161689890267136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33671087 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936ê5”ö2š- _wandbruntime‚2124  _runtime‚2124E§wóÙ&›ÓM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76333261 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33678308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171850053529600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42654953 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171594854613790 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639110601872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42664944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42654953 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174498512865792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164460912092672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10202237 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161724826685440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2648442296832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2648354352640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42664944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30909310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76343254 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚363 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174498407882240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171850070568960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168955830149120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2736110310912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168955847188480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171850070568960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2736284510720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784370182718965e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2648442296832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76333261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76343254 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2639024464670 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚4499 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33678310 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30909066 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171594957790352 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20706829 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161724801781760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2648354352640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33678308 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 _runtime‚2124.163047802 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296—å3eÑœØM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171594957790352 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20706829 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161724801781760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2648354352640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33678308 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 _runtime‚2124.163047802 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76333261 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33678308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171850053529600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42654953 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171594854613790 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639110601872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42664944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42654953 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174498512865792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164460912092672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10202237 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161724826685440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2648442296832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2648354352640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42664944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30909310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76343254 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDevit!mò ceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚363 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174498407882240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171850070568960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168955830149120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2736110310912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168955847188480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171850070568960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2736284510720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784370182718965e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2648442296832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76333261 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76343254 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2639024464670 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚4499 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33678310 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30909066ëôôþì2- _wandbruntime‚2124  _runtime‚2124aéÇÀž€ 2loss/SRqqt1/assignment_loss‚0.6115918208908458 1loss/SRqqt2/assignment_loss‚0.534897048191407 2loss/FBt1/assignment_loss‚0.004057503324875775 2loss/FBt2/assignment_loss‚0.015149269426750222 /loss/FRt2/detection_loss‚1.5766661905655914 0loss/SRqqt2/detection_loss‚0.676350317815126 /loss/FBt1/detection_loss‚0.7531254761428996 trainer/global_step‚4499 0loss/FRt1/assignment_loss‚1.7557403550103858 0loss/FRt2/assignment_loss‚0.8572975043022987 0loss/SRqqt1/detection_loss‚1.024843773899854 $ _timestamp‚1.784370182720819e+09 _runtime‚2124.164353276 0loss/FBt2/detection_loss‚0.45771851737550656 %loss/total_loss‚8.890489523516584 epoch‚5 /loss/FRt1/detection_loss‚0.6230517465710431 _step‚364-Z”Ì Ÿ… 0loss/FBt2/detection_loss‚0.45771851737550656 %loss/total_loss‚8.890489523516584 epoch‚5 /loss/FRt1/detection_loss‚0.6230517465710431 _step‚364 2loss/SRqqt1/assignment_loss‚0.6115918208908458 1loss/SRqqt2/assignment_loss‚0.534897048191407 2loss/FBt1/assignment_loss‚0.004057503324875775 2loss/FBt2/assignment_loss‚0.015149269426750222 /loss/FRt2/detection_loss‚1.5766661905655914 0loss/SRqqt2/detection_loss‚0.676350317815126 /loss/FBt1/detection_loss‚0.7531254761428996 trainer/global_step‚4499 0loss/FRt1/assignment_loss‚1.7557403550103858 0loss/FRt2/assignment_loss‚0.8572975043022987 0loss/SRqqt1/detection_loss‚1.024843773899854 $ _timestamp‚1.784370182720819e+09 _runtime‚2124.164353276쨳íi‘ ‚(:† Œ¨íÒ—Ú‰Œ gpu.0.gpu‚63.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚176.679' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÐ.ðÚ¡‚(:Ï Œ¨íÒèÇðŽ disk.sda.out‚ 52.01171875( proc.memory.availableMB‚ 218.20703125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 2905.93359375 disk.sda2.in‚0 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚ 52.01171875 disk.sda3.in‚ 0.10546875 proc.cpu.threads‚17* disk./.usagePercent‚39.614876704313836 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.out‚4350.525390625$ memory_percent‚98.66816997528076$ disk./.usageGB‚590.2902908325195# disk.nvme0c0n1.in‚ 2905.93359375 disk.sda1.out‚0* proc.memory.percent‚15.922880172729492 disk.sda2.out‚0 disk.dm-0.in‚ 2905.93359375 disk.sda3.out‚ 52.01171875" proc.memory.rssMB‚ 2608.8046875 disk.drbd2165.in‚0# disk.nvme0n1.out‚4350.525390625 cpu‚26.914233716829475 network.sent‚ 5.900338e+06 network.recv‚701554 disk.rbd1.in‚0 disk.rbd1.out‚5.1875% disk.nvme0c0n1.out‚4350.525390625 disk.sda.in‚ 0.10546875ŠM¥¢‚(:… ›¨íÒºßÕ‹ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚186.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“¶¬Ö£‚(:Ë ›¨íÒ­© disk.drbd2165.in‚0 disk.rbd1.out‚5.1875( proc.memory.availableMB‚ 218.01953125' disk.nvme0c0n1.out‚4355.52099609375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.10546875$ memory_percent‚98.66931438446045% disk.nvme0n1.out‚4355.52099609375 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚26.90957949909414 network.sent‚ 5.963073e+06) disk./.usagePercent‚39.61479785399329$ disk./.usageGB‚590.2891159057617" disk.dm-0.out‚4355.52099609375 disk.sda.in‚ 0.10546875# disk.nvme0c0n1.in‚ 2905.93359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2905.93359375 disk.sda3.out‚52.15625 disk.rbd1.in‚0* proc.memory.percent‚15.923666954040527 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2608.93359375! disk.nvme0n1.in‚ 2905.93359375 disk.dm-1.in‚ 0.10546875 disk.dm-1.out‚52.15625 proc.cpu.threads‚17 network.recv‚706927 disk.sda.out‚52.15625¯}÷2¤- _wandbruntime‚2145  _runtime‚2145ëÂ%1Ä¥¾ _runtime‚2145.653907019 _step‚365 % lr-AdamW/pg1‚0.000829122067805972 % lr-AdamW/pg2‚0.000829122067805972 trainer/global_step‚4549 $ _timestamp‚1.784370204210047e+09¦Bɦà _runtime‚2145.653907019 _step‚365 % lr-AdamW/pg1‚0.000829122067805972 % lr-AdamW/pg2‚0.000829122067805972 trainer/global_step‚4549 $ _timestamp‚1.784370204210047e+09íÛ nH2§- _wandbruntime‚2145  _runtime‚2145&«¥Ð'¨ÊO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43111574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2666855122848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173668507815936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2676373155328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2666941259938 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176344880971264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166200669490688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173668490776576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170742959880192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34032137 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77153690 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173409918179490 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20926548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚4549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163435465998336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43121551 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2765178588672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170742976919552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176344775994880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10307623 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34032139 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43111574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 _step‚366 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2676285218304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2676285218304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10307425 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173409815003040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176344775994880 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34032139 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144ÅÀãÕ'©ÏO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176344880971264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31234171 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20926545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173668507815936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173668490776576 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34032137 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77143711 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166200480865792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2676373155328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31233970 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 % _timestamp‚1.7843702042112472e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2765014867456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43121551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 _runtime‚2145.658614904 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170742976919552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176344775994880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10307623 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34032139 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43111574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 _step‚366 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2676285218304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2676285218304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10307425 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173409815003040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176344775994880 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34032139 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43111574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2666855122848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173668507815936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2676373155328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2666941259938 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176344880971264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166200669490688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173668490776576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170742959880192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34032137 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77153690 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173409918179490 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20926548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚4549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163435465998336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43121551 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2765178588672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43130846 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34039358 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170779431915520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166235977553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77160213 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2667422444378 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDey–‰³3viceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176382561302016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176382456318464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77170206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10309807 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚367 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2765600040448 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10309583 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173446957536460 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31240612 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2676854577664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163470377512960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2765770045952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2667508581580 _runtime‚2145.998700149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77160213 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¤ébªß&¬ÙM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43130846 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34039358 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170779431915520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166235977553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77160213 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2667422444378 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176382561302016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176382456318464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77170206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10309807 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚367 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2765600040448 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10309583 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173446957536460 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31240612 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2676854577664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163470377512960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2765770045952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2667508581580 _runtime‚2145.998700149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77160213 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2676942521856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20931032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163470402416640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166236172462592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173705601740800ï¢?H,2­- _wandbruntime‚2145  _runtime‚2145þ•Q® 0loss/FRt1/assignment_loss‚1.8321523314579986 2loss/FBt1/assignment_loss‚0.017235798238312613 /loss/FBt2/detection_loss‚0.5699936678674795 trainer/global_step‚4549 $ _timestamp‚1.784370204556317e+09 _runtime‚ 2145.99985026 _step‚368 2loss/FBt2/assignment_loss‚0.001686466086562665 /loss/FRt2/detection_loss‚1.5262090784088789 1loss/SRqqt1/detection_loss‚1.0166806819480518 2loss/SRqqt2/assignment_loss‚0.6006086762076556 1loss/SRqqt2/detection_loss‚0.6982386222861092 %loss/total_loss‚9.269145051562159 epoch‚5 0loss/FRt2/assignment_loss‚0.7360719345274015 2loss/SRqqt1/assignment_loss‚0.7595322535855176 /loss/FRt1/detection_loss‚0.5887640965837154 /loss/FBt1/detection_loss‚0.9219714443644752Ô´¿á ¯† 2loss/FBt1/assignment_loss‚0.017235798238312613 /loss/FBt2/detection_loss‚0.5699936678674795 trainer/global_step‚4549 $ _timestamp‚1.784370204556317e+09 _runtime‚ 2145.99985026 _step‚368 2loss/FBt2/assignment_loss‚0.001686466086562665 /loss/FRt2/detection_loss‚1.5262090784088789 1loss/SRqqt1/detection_loss‚1.0166806819480518 2loss/SRqqt2/assignment_loss‚0.6006086762076556 1loss/SRqqt2/detection_loss‚0.6982386222861092 %loss/total_loss‚9.269145051562159 epoch‚5 0loss/FRt2/assignment_loss‚0.7360719345274015 2loss/SRqqt1/assignment_loss‚0.7595322535855176 /loss/FRt1/detection_loss‚0.5887640965837154 /loss/FBt1/detection_loss‚0.9219714443644752 0loss/FRt1/assignment_loss‚1.8321523314579986ðŠ””Ê‘°‚(:† ª¨íÒØÊአgpu.0.gpu‚49.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚123.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Nß±‚(:Ô ª¨íÒ­’±Ž* disk./.usagePercent‚39.614811422392606 disk.drbd2165.out‚0% disk.nvme0n1.out‚4358.06884765625 disk.sda2.in‚0 disk.sda3.out‚ 52.46484375 disk.rbd1.in‚0" disk.dm-0.out‚4358.06884765625$ memory_percent‚98.66936206817627 network.sent‚ 5.967889e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2905.93359375* proc.memory.percent‚15.923666954040527 proc.cpu.threads‚17 disk.rbd0.out‚0# proc.memory.rssMB‚ 2608.93359375$ disk./.usageGB‚590.2893180847168! disk.nvme0n1.in‚ 2905.93359375 disk.sda1.out‚0 disk.rbd1.out‚5.25' disk.nvme0c0n1.out‚4358.06884765625 disk.dm-1.in‚ 0.10546875 cpu‚26.905023720142896 network.recv‚709728 disk.sda.in‚ 0.10546875 disk.sda.out‚ 52.46484375 disk.rbd0.in‚0 disk.dm-1.out‚ 52.46484375 disk.dm-0.in‚ 2905.93359375( proc.memory.availableMB‚ 218.01171875e;Hý2²- _wandbruntime‚2167  _runtime‚2167ºÿƳÀ _runtime‚2167.605969623 _step‚369 & lr-AdamW/pg1‚0.0008286173202353497 & lr-AdamW/pg2‚0.0008286173202353497 trainer/global_step‚4599 $ _timestamp‚1.784370226162159e+09•Õ«Ë´Å _step‚369 & lr-AdamW/pg1‚0.0008286173202353497 & lr-AdamW/pg2‚0.0008286173202353497 trainer/global_step‚4599 $ _timestamp‚1.784370226162159e+09 _runtime‚2167.605969623ñžú"ç2µ- _wandbruntime‚2167  _runtime‚2167µF -Ð'¶ÊO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34393189 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚175524056027136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚178228824431104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172566561646592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚178228929407488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172566578685952 trainer/global_step‚4599 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2794531950592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178228929407488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175524038987776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167975573680128 % _timestamp‚1.7843702261632066e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34393187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175524056027136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43587453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21150745 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21150748 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175261814749148 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415855 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2704873380352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165181041729536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43577476 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2704785443328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34393187 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2704785443328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77980642 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2695339239646 _runtime‚2167.610646662 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10416058 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2794689380352 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178228824431104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43577476 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31566600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167975756013568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2695253102556 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2704873380352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146!±Þÿ ·ÏO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2704873380352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34393189 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚175524056027136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚178228824431104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172566561646592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚178228929407488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172566578685952 trainer/global_step‚4599 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2794531950592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178228929407488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175524038987776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167975573680128 % _timestamp‚1.7843702261632066e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34393187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175524056027136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43587453 LFDÏçžÚ¼eviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21150745 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21150748 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175261814749148 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415855 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2704873380352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165181041729536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43577476 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2704785443328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34393187 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2704785443328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77980642 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2695339239646 _runtime‚2167.610646662 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10416058 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2794689380352 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178228824431104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43577476 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31566600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167975756013568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2695253102556 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146òn™2¸- _wandbruntime‚2167  _runtime‚2167ˆ@F_Ú&¹ÔM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2705354802688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175298957282568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43596748 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10417981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2705354802688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚215 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168011070963712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2795117719552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175561149952000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31573210 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175298854106006 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31573428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77987165 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43596748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10418196 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168011263775744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77997158 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34400408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77997158 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175561149952000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚178266609738240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _runtime‚2167.950276213 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21155232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165215953244160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34400410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34400410 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2705442746880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2795285627904 % _timestamp‚1.7843702265061336e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21155229 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2705442746880 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43586757 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2695906561288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _step‚371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚218 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2695820424086 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34400408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178266504754688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165215978147840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚178266609738240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175561166991360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77987165 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172603033681920 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192EÃÄâß&ºÙM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43586757 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2695906561288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _step‚371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚218 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2695820424086 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34400408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178266504754688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165215978147840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚178266609738240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175561166991360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77987165 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172603033681920 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2705354802688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175298957282568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43596748 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10417981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2705354802688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚215 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168011070963712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2795117719552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175561149952000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31573210 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175298854106006 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31573428 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77987165 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43596748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10418196 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168011263775744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77997158 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34400408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77997158 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175561149952000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚178266609738240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _runtime‚2167.950276213 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21155232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165215953244160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34400410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34400410 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2705442746880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2795285627904 % _timestamp‚1.7843702265061336e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21155229 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2705442746880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactiveéT«€_split.all.allocated‚31899830 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10524882 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34754239 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177379587198976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177379604238336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166926617460736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2723737219354 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177379604238336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44053355 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166926642364416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44053355 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177113917671706 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174390163412992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚2189.394888369 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180112977843712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2723651082264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2733285668352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44043378 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169750837482496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180112872867328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2733373605376 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177113814495256 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34754239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21374945 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44043378 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180112872867328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2824195118080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2824018813952 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31899830 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10524882 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34754239 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177379587198976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177379604238336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166926617460736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2723737219354 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177379604238336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44053355 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166926642364416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44053355 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177113917671706 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174390163412992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚2189.394888369 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180112977843712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2723651082264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2733285668352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44043378 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169750837482496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180112872867328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2733373605376 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177113814495256 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34754239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21374945 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44043378 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180112872867328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2824195118080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2824018813952 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34761458 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21379432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169786337570816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2724304540996 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _runtime‚2189.734369069 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚4649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78824110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚215 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177416715202560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78824110 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34761460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177416715202560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10526936 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31906153 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177150957028676 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44062650 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _step‚375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180150553190912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2824783691776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0p&~¡}ÈÙM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177416715202560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78824110 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34761460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177416715202560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10526936 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31906153 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177150957028676 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44062650 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _step‚375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180150553190912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2824783691776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44062650 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44052659 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180150658174464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177416698163200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174426635448320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2724218403794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34761458 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21379432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169786337570816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2724304540996 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _runtime‚2189.734369069 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚4649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78824110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚215 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452÷ çØ2É- _wandbruntime‚2189  _runtime‚2189™ b Ê€ 1loss/SRqqt1/detection_loss‚0.9278435956188885 /loss/FBt2/detection_loss‚0.6819669588019704 0loss/FRt1/assignment_loss‚1.7797114919827606 2loss/FBt1/assignment_loss‚0.004567930808291543 .loss/FRt2/detection_loss‚1.589663552266141 0loss/SRqqt2/detection_loss‚0.742338378481104 _step‚376 %loss/total_loss‚8.905371353624023 epoch‚5 _runtime‚2189.735878839 0loss/FRt2/assignment_loss‚0.7273957557750471 2loss/SRqqt2/assignment_loss‚0.5062177501931082 /loss/FRt1/detection_loss‚0.6193917991087999 /loss/FBt1/detection_loss‚0.5006696174386889 trainer/global_step‚4649 % _timestamp‚1.7843702482922034e+09 2loss/SRqqt1/assignment_loss‚0.7475890608795185 1loss/FBt2/assignment_loss‚0.07801546226970461JŽ|² Ë… trainer/global_step‚4649 % _timestamp‚1.7843702482922034e+09 2loss/SRqqt1/assignment_loss‚0.7475890608795185 1loss/FBt2/assignment_loss‚0.07801546226970461 1loss/SRqqt1/detection_loss‚0.9278435956188885 /loss/FBt2/detection_loss‚0.6819669588019704 0loss/FRt1/assignment_loss‚1.7797114919827606 2loss/FBt1/assignment_loss‚0.004567930808291543 .loss/FRt2/detection_loss‚1.589663552266141 0loss/SRqqt2/detection_loss‚0.742338378481104 _step‚376 %loss/total_loss‚8.905371353624023 epoch‚5 _runtime‚2189.735878839 0loss/FRt2/assignment_loss‚0.7273957557750471 2loss/SRqqt2/assignment_loss‚0.5062177501931082 /loss/FRt1/detection_loss‚0.6193917991087999 /loss/FBt1/detection_loss‚0.5006696174386889øFßxœÌ‚(:‘ ȨíÒëà‹ gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚161.348' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.269600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÊc¿Í‚(:´ ȨíÒã«… disk.rbd0.out‚0 disk.sda.out‚55$ memory_percent‚98.66924285888672 disk.rbd1.out‚5.25( disk./.usagePercent‚39.6146311930885$ disk./.usageGB‚590.2866325378418 disk.sda3.in‚ 0.10546875 disk.sda3.out‚55 disk.sda2.out‚0 disk.dm-1.out‚55% proc.memory.availableMB‚ 218.03125# proc.memory.rssMB‚ 2608.93359375* proc.memory.percent‚15.923666954040527 cpu‚26.896084711283624 disk.sda.in‚ 0.10546875! disk.nvme0n1.in‚ 3010.76953125 disk.sda1.out‚0" disk.dm-0.out‚4376.74267578125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 3010.76953125 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 3010.76953125 network.recv‚720076 network.sent‚ 6.092451e+06% disk.nvme0n1.out‚4376.74267578125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚4376.74267578125 disk.drbd2165.in‚0 disk.sda1.in‚0î:˜‘΂(:† רíÒÛƼŠ gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚184.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-`ÙÏ‚(:Î רíÒʬ‡ disk.sda3.out‚ 55.54296875 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚39.61464271342754! disk.nvme0n1.in‚ 3010.79296875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.10546875 disk.dm-0.in‚ 3010.79296875" disk.dm-0.out‚4385.04052734375& proc.memory.availableMB‚ 218.015625 cpu‚26.891743156162082$ disk./.usageGB‚590.2868041992188 disk.sda.in‚ 0.10546875 disk.sda2.in‚0 disk.dm-1.out‚ 55.54296875 network.sent‚ 6.155079e+06 disk.rbd1.out‚5.375 proc.cpu.threads‚17 network.recv‚725625# disk.nvme0c0n1.in‚ 3010.79296875' disk.nvme0c0n1.out‚4385.04052734375$ memory_percent‚98.66933822631836 proc.memory.rssMB‚ 2609.0625 disk.sda.out‚ 55.54296875 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0* proc.memory.percent‚15.924453735351562 disk.sda3.in‚ 0.10546875 disk.drbd2165.in‚0% disk.nvme0n1.out‚4385.04052734375 disk.sda1.out‚0è3­2Ð- _wandbruntime‚2211  _runtime‚2211ó®È°ÇÑÁ & lr-AdamW/pg1‚0.0008275883924382785 & lr-AdamW/pg2‚0.0008275883924382785 trainer/global_step‚4699 % _timestamp‚1.7843702699620447e+09 _runtime‚2211.405871516 _step‚377Ž§ ŒÌÒÆ _step‚377 & lr-AdamW/pg1‚0.0008275883924382785 & lr-AdamW/pg2‚0.0008275883924382785 trainer/global_step‚4699 % _timestamp‚1.7843702699620447e+09 _runtime‚2211.405871516ùœêªç2Ó- _wandbruntime‚2211  _runtime‚2211ÁŸdsÎ'ÔÈO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181996921303552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179235135410176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171525718927360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21599148 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21599145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168672193191936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2761873830400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2761873830400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚378 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168672218095616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 # _timestamp‚1.78437026996314e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178965917417814 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2853702039552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181997026279936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179235152449536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2761785893376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2853525735424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2752135199062 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936ƤÓ'ÕÍO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32232285 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35115289 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚176213782218752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181997026279936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2752049061972 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179235152449536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44509280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2761785893376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10633363 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚223 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178965814241364 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176213765179392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4699 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44519257 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35115289 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35115287 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44519257 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _runtime‚2211.407128238 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181996921303552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171525920135168 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32232511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 # _timestamp‚1.78437026996314e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178965917417814 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2853702039552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181997026279936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179235152449536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2761785893376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2853525735424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2752135199062 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181996921303552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179235135410176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171525718927360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21599148 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21599145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168672193191936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2761873830400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2761873830400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚378 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168672218095616 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_e æÞ†~nd/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚379 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44518561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171561421329408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44518561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35122510 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44528552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 % _timestamp‚1.7843702703094072e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182034706610688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35122508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21603629 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179002853598222 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179272246374400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179002956774784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182034706610688 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79651062 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176250254254080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2752702520704 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79641069 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35122508 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179272246374400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2762355252736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32239194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182034601627136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44528552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176250237214720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2762443196928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32238961 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10635332 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452ºûÉ“ß&ØÙM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179272246374400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179002956774784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182034706610688 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79651062 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176250254254080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2752702520704 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79641069 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35122508 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179272246374400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2762355252736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32239194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182034601627136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44528552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176250237214720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2762443196928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32238961 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10635332 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179272263413760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179272263413760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2854291719168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2752616383502 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _step‚379 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44518561 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171561421329408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44518561 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35122510 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44528552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 % _timestamp‚1.7843702703094072e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182034706610688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35122508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21603629 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179002853598222ûåߌƒ2Ù- _wandbruntime‚2211  _runtime‚2211i% -Ú $ _timestamp‚1.784370270311209e+09 _step‚380 epoch‚5 2loss/SRqqt2/assignment_loss‚0.5695944563992595 2loss/FBt2/assignment_loss‚0.000427830273732314 .loss/FRt2/detection_loss‚1.659325551688291 /loss/FBt2/detection_loss‚0.5270750249942971 trainer/global_step‚4699 2loss/SRqqt1/assignment_loss‚0.6851851098112453 /loss/FRt1/detection_loss‚0.5489564685255269 /loss/FBt1/detection_loss‚1.0041978673636913 %loss/total_loss‚9.115164252108297 0loss/FRt2/assignment_loss‚0.8271773214242897 2loss/FBt1/assignment_loss‚0.018962189514396356 1loss/SRqqt1/detection_loss‚0.7408536769204087 1loss/SRqqt2/detection_loss‚0.6536109711745525 _runtime‚2211.754770397 0loss/FRt1/assignment_loss‚1.8797977840186058.`4Å Û† 2loss/SRqqt1/assignment_loss‚0.6851851098112453 /loss/FRt1/detection_loss‚0.5489564685255269 /loss/FBt1/detection_loss‚1.0041978673636913 %loss/total_loss‚9.115164252108297 0loss/FRt2/assignment_loss‚0.8271773214242897 2loss/FBt1/assignment_loss‚0.018962189514396356 1loss/SRqqt1/detection_loss‚0.7408536769204087 1loss/SRqqt2/detection_loss‚0.6536109711745525 _runtime‚2211.754770397 0loss/FRt1/assignment_loss‚1.8797977840186058 $ _timestamp‚1.784370270311209e+09 _step‚380 epoch‚5 2loss/SRqqt2/assignment_loss‚0.5695944563992595 2loss/FBt2/assignment_loss‚0.000427830273732314 .loss/FRt2/detection_loss‚1.659325551688291 /loss/FBt2/detection_loss‚0.5270750249942971 trainer/global_step‚4699ühN¾BœÜ‚(:‘ æ¨íÒàŽÛ‹ gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚181.806' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.361200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H(ñ2ÎÝ‚(:à æ¨íÒ’¸‚ disk.sda.out‚ 55.859375$ memory_percent‚98.67241382598877 network.recv‚728986 disk.dm-1.in‚ 0.10546875" disk.dm-0.out‚4394.58837890625 proc.cpu.threads‚17 network.sent‚ 6.173096e+06 disk.sda3.in‚ 0.10546875 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 3010.796875% disk.nvme0n1.out‚4394.58837890625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚26.88760892211041) disk./.usagePercent‚39.61459048789055 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚4394.58837890625 disk.sda2.in‚0 disk.dm-0.in‚ 3010.796875 proc.memory.rssMB‚ 2609.0625* proc.memory.percent‚15.924453735351562 disk.rbd1.out‚5.375 disk.nvme0n1.in‚ 3010.796875 disk.dm-1.out‚ 55.859375( proc.memory.availableMB‚ 217.51171875$ disk./.usageGB‚590.2860260009766 disk.sda.in‚ 0.10546875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 55.859375AjÔk2Þ- _wandbruntime‚2233  _runtime‚2233(—K¿ÇßÁ trainer/global_step‚4749 % _timestamp‚1.7843702917588346e+09 _runtime‚2233.202663555 _step‚381 & lr-AdamW/pg1‚0.0008270642286921968 & lr-AdamW/pg2‚0.0008270642286921968o—UêÌàÆ trainer/global_step‚4749 % _timestamp‚1.7843702917588346e+09 _runtime‚2233.202663555 _step‚381 & lr-AdamW/pg1‚0.0008270642286921968 & lr-AdamW/pg2‚0.0008270642286921968ýb¤è¢2á- _wandbruntime‚2233  _runtime‚2233ž1KƒÐ'âÊO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181090683621376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183880969739776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32566172 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44975182 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44985159 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843702917599514e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21823348 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10742827 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183881074716160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173300759331328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2790374055424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44985159 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181090700660736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064áXèÒãÏO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173300759331328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2790374055424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44985159 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181090700660736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDevi~åW¹$ceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181090683621376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183880969739776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32566172 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44975182 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44985159 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843702917599514e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21823348 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10742827 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183881074716160þþ=C}2ä- _wandbruntime‚2233  _runtime‚2233ʬ0‹Ú&åÔM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44994454 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32572796 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181127811624960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2883572357632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180854853344330 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2883757043200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44984463 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80468021 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80478014 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80468021 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178073838981120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2790855477760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21827829 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170452680437760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚4749 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35483558 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2790943421952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173336462384640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _runtime‚2233.545540346 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181127811624960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2790943421952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2790855477760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2781014363210 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 % _timestamp‚1.7843702921013339e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173336252795392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183918650063360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80478014 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35483560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35483560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178073856020480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Ø¢nß&æÙM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181127794585600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2781100500412 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _runtime‚2233.545540346 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181127811624960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2790943421952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2790855477760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2781014363210 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 % _timestamp‚1.7843702921013339e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173336252795392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183918650063360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80478014 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35483560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35483560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178073856020480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44994454 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32572796 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181127811624960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2883572357632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180854853344330 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2883757043200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44984463 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80468021 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80478014 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80468021 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178073838981120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2790855477760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21827829 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170452680437760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚4749 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35483558 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2790943421952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173336462384640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10852009 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35837389 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22047545 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182946248871936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185765123152384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2818786343424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10851785 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2808845021388 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182669916910030 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182946231832576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35837389 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 _runtime‚ 2255.25962145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81288450 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175075919772160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4799 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172163369558016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81278471 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185765123152384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2912371812864 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185765018176000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182946248871936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172163344654336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784370313812076e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35837387 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45451061 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0N®Ó'óÍO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45441084 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10852009 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35837389 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22047545 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182946248871936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185765123152384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2818786343424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10851785 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2808845021388 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182669916910030 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182946231832576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35837389 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 _runtime‚ 2255.25962145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81288450 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175075919772160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4799 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172163369558016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81278471 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185765123152384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2912371812864 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185765018176000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182946248871936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172163344654336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784370313812076e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35837387 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45451061 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32899330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2818874280448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024‚#{Û(2ô- _wandbruntime‚2255  _runtime‚2255ê7Ä}Ú&õÔM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182983359836160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2912965066240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81294973 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81304966 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32905887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175111221235200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚387 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚2255.594690864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182983359836160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185802803483136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2819355702784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185802803483136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179897440747520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32906111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45460356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185802698499584 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22052029 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35844610 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10853858 trainer/global_step‚4799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81294973 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2819443646976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182983342796800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35844610 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22052032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172198256168960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2809412342918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81304966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45460356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185802698499584 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22052029 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35844610 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10853858 trainer/global_step‚4799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81294973 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2819443646976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182983342796800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35844610 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22052032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172198256168960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2809412342918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81304966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182983359836160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2912965066240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81294973 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81304966 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32905887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175111221235200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚387 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚2255.594690864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182983359836160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185802803483136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2819355702784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185802803483136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179897440747520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32906111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23828333616256714 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12565524876117706 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956779479980469 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08011797070503235 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 'particle/f_score‚0.6428202390670776 7validation_average_jet_accuracy‚0.34816351532936096 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6333394050598145 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949368238449097 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44528448581695557 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01917693018913269 *jet/accuracy_1_of_3‚0.9207865595817566 *jet/accuracy_2_of_4‚0.7814957499504089 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6921564340591431 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4489705562591553 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14110557734966278 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20872433483600616 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2063622921705246 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988896250724792 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947029948234558 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1393243670463562 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03597324341535568 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.727642834186554 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6534057855606079 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29875797033309937 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8308897614479065 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27881062030792236 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12621884047985077 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7377318739891052 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964994788169861 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6459765434265137 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12510527670383453 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7403329610824585 *jet/accuracy_1_of_2‚0.7851064801216125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6892697215080261 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.61750727891922 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10850759595632553 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8149122595787048 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7490637898445129 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.459747850894928 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07877687364816666 (jet/accuracy_2_of_3‚0.52955561876297 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977381825447083 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.6267485618591309 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07932370901107788 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09884699434041977 particle/accuracy_1_of_4‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14701016247272491 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46224889159202576 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7748293280601501 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1353786140680313 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 epoch‚5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6512166261672974 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45075753331184387 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009758885018527508 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6363818645477295 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035687971860170364 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6574456691741943 /particle/accuracy_3_of_3‚0.9989849328994751 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40432095527648926 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8983155488967896 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7426910996437073 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7449464797973633 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6912540197372437 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328112006187439 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11604263633489609 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6947249174118042 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6327370405197144 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3826625347137451 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.653264045715332 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7632619142532349 +jet/accuracy_2_of_2‚0.14839962124824524 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6480942964553833 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07861310988664627 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5799540281295776 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 )loss/val_total_loss‚8.626643459919654 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17103905975818634 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989315390586853 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15527240931987762 )jet/accuracy_1_of_1‚0.298084020614624 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7431623339653015 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8149122595787048 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193894386291504 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5539460778236389 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7638938426971436 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20244181156158447 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6496102809906006 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6314288377761841 _runtime‚2278.982006643 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37943005561828613 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1490381509065628 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1646101027727127 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5507342219352722 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7051049470901489 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1511908769607544 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11907421797513962 +particle/specificity‚0.9116431474685669 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46636441349983215 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17031250894069672 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985410571098328 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13593687117099762 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 ,jet/accuracy_4_of_4‚0.014682539738714695 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10287635773420334 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 /particle/accuracy_3_of_4‚0.9943783283233643 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8268925547599792 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5982388257980347 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.631013035774231 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8230769634246826 % _timestamp‚1.7843703375374749e+09 _step‚389 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963453412055969 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7975679039955139 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10314363241195679 (particle/accuracy‚0.8504619598388672 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3888888657093048 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967328310012817 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14330673217773438 /particle/accuracy_4_of_4‚0.9506038427352905 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7021610736846924 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1ùD‹Å"þ¸{ 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8983155488967896 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7426910996437073 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7449464797973633 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6912540197372437 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9328112006187439 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11604263633489609 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6947249174118042 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6327370405197144 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3826625347137451 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.653264045715332 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7632619142532349 +jet/accuracy_2_of_2‚0.14839962124824524 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6480942964553833 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07861310988664627 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5799540281295776 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 )loss/val_total_loss‚8.626643459919654 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17103905975818634 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989315390586853 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15527240931987762 )jet/accuracy_1_of_1‚0.298084020614624 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7431623339653015 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8149122595787048 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193894386291504 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5539460778236389 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7638938426971436 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20244181156158447 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6496102809906006 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6314288377761841 _runtime‚2278.982006643 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37943005561828613 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1490381509065628 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1646101027727127 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5507342219352722 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7051049470901489 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1511908769607544 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11907421797513962 +particle/specificity‚0.9116431474685669 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46636441349983215 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17031250894069672 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985410571098328 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13593687117099762 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 ,jet/accuracy_4_of_4‚0.014682539738714695 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10287635773420334 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 /particle/accuracy_3_of_4‚0.9943783283233643 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8268925547599792 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5982388257980347 @&Purity/1FRt*SRqq¢JHvœ*t2FBt/event_proportion‚0.0027204242069274187 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.631013035774231 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8230769634246826 % _timestamp‚1.7843703375374749e+09 _step‚389 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963453412055969 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7975679039955139 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10314363241195679 (particle/accuracy‚0.8504619598388672 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3888888657093048 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967328310012817 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14330673217773438 /particle/accuracy_4_of_4‚0.9506038427352905 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7021610736846924 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.699691653251648 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6764088869094849 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02406766265630722 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8707548975944519 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14918652176856995 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6638686656951904 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1297488808631897 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6512166261672974 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7225517630577087 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7749150395393372 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07351233065128326 +jet/accuracy_3_of_3‚0.07366122305393219 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876849055290222 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4925127923488617 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5283854007720947 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01862558349967003 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07351233065128326 )jet/accuracy_1_of_4‚0.953079342842102 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 trainer/global_step‚4805 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13478806614875793 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7975679039955139 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6319444179534912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8672921061515808 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13478806614875793 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23828333616256714 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12565524876117706 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956779479980469 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08011797070503235 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 'particle/f_score‚0.6428202390670776 7validation_average_jet_accuracy‚0.34816351532936096 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6333394050598145 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949368238449097 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44528448581695557 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01917693018913269 *jet/accuracy_1_of_3‚0.9207865595817566 *jet/accuracy_2_of_4‚0.7814957499504089 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6921564340591431 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4489705562591553 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14110557734966278 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20872433483600616 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2063622921705246 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988896250724792 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947029948234558 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1393243670463562 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03597324341535568 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.727642834186554 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6534057855606079 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29875797033309937 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8308897614479065 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27881062030792236 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12621884047985077 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7377318739891052 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964994788169861 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6459765434265137 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12510527670383453 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7403329610824585 *jet/accuracy_1_of_2‚0.7851064801216125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6892697215080261 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.61750727891922 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10850759595632553 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8149122595787048 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7490637898445129 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.459747850894928 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07877687364816666 (jet/accuracy_2_of_3‚0.52955561876297 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977381825447083 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.6267485618591309 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07932370901107788 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09884699434041977 particle/accuracy_1_of_4‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14701016247272491 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46224889159202576 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7748293280601501 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1353786140680313 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 epoch‚5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6512166261672974 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45075753331184387 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009758885018527508 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6363818645477295 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035687971860170364 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6574456691741943 /particle/accuracy_3_of_3‚0.9989849328994751 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40432095527648926 ("Purity/1FRt2SRqqt1FBt/event_purity‚0…ÙJ“Óÿ‚(:… ¢©íÒ™£³Š gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02aÃÊ€‚(:¿ ¢©íÒÙãæŒ disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 57.609375 disk.dm-0.in‚ 3124.046875" proc.memory.rssMB‚ 2739.3359375* proc.memory.percent‚16.719579696655273 disk.sda.out‚ 57.609375! disk.nvme0c0n1.in‚ 3124.046875 disk.nvme0n1.in‚ 3124.046875 disk.sda1.out‚0 cpu‚26.915742621465338% disk.nvme0n1.out‚4422.55908203125# memory_percent‚99.1326093673706 disk.sda.in‚0.109375 disk.dm-1.in‚0.109375 proc.cpu.threads‚17 disk.sda3.out‚ 57.609375 disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚4422.55908203125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚4422.55908203125( proc.memory.availableMB‚ 142.11328125 network.sent‚ 6.353781e+06 network.recv‚748290$ disk./.usageGB‚590.2845153808594 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.out‚5.5 disk.sda1.in‚0* disk./.usagePercent‚39.614489108906994 disk.rbd0.in‚0Í T½‘‚(:† ±©íÒÖ³½Š gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚181.488' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09΂–­‚‚(:¢ ±©íÒÏÛïŒ network.sent‚ 6.411033e+06 disk.drbd2165.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚590.2842102050781 disk.sda3.in‚0.109375 disk.dm-1.out‚57.875# memory_percent‚98.7999677658081" proc.memory.rssMB‚ 2684.7890625 proc.cpu.threads‚17( proc.memory.availableMB‚ 196.61328125 disk.rbd1.out‚5.5625 disk.nvme0n1.out‚ 4555.78125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0) disk./.usagePercent‚39.61446862830426! disk.nvme0c0n1.out‚ 4555.78125 disk.sda.in‚0.109375 disk.nvme0c0n1.in‚ 3219.59375 disk.sda1.in‚0 disk.dm-0.out‚ 4555.78125 network.recv‚753413 disk.sda3.out‚57.875 disk.sda.out‚57.875 disk.sda1.out‚0) proc.memory.percent‚16.38665199279785 disk.dm-0.in‚ 3219.59375 disk.nvme0n1.in‚ 3219.59375 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0 cpu‚26.918868751983176ß·Eå2ƒ- _wandbruntime‚2299  _runtime‚2299‚˜ß¦Æ„À _runtime‚2299.531588635 _step‚390 & lr-AdamW/pg1‚0.0008259965434825384 & lr-AdamW/pg2‚0.0008259965434825384 trainer/global_step‚4849 $ _timestamp‚1.784370358087735e+09n×yIË…Å _runtime‚2299.531588635 _step‚390 & lr-AdamW/pg1‚0.0008259965434825384 & lr-AdamW/pg2‚0.0008259965434825384 trainer/global_step‚4849 $ _timestamp‚1.784370358087735e+09†C.î:2†- _wandbruntime‚2299  _runtime‚2299ïb ›Ð'‡ÊO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46060541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188626559823360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33408959 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22376328 trainer/global_step‚4849 _runtime‚2299.533047029 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2863888363008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36432965 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2958620505088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185495519955054 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚185762776436736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚188626559823360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46060541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2958440006656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2863888363008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2853817739632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46050564 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82483529 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33408723 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11032631 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182641788352512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2853731602542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174823433809920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚188626664799744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2863888363008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36432965 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2958620505088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185495519955054 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚185762776436736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚188626559823360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46060541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2958440006656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2863888363008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2853817739632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46050564 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82483529 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33408723 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11032631 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182641788352512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2853731602542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174823433809920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚188626664799744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188626559823360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0‡¦‚c2‰- _wandbruntime‚2299  _runtime‚2299kA³`Ú&ŠÔM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22380809 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36440188 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚188664345130496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2959018421248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185532559311912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82510024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182678260387840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33415307 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2959207301120 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚4849 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185799870361600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46069836 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2299.874912881 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22380812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36440188 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177817363745792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11034744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82500031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188879872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185799887400960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185532662488474 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188664240146944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46059845 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177817577529344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2854298924072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82510024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11034498 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2864369785344ñw Ðß&‹ÙM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33415556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚188664345130496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36440186 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182678260387840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33415307 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2959207301120 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚4849 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185799870361600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46069836 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2299.874912881 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22380812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36440188 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177817363745792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11034744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82500031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188879872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185799887400960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185532662488474 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188664240146944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46059845 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177817577529344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2854298924072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82510024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11034498 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2864369785344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22380809 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36440188 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚188664345130496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2959018421248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185532559311912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82510024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46516466 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187618324647936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2987860011008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4899 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2892300651008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46526443 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36794017 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚2322.034735382 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190510608259584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11142234 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46516466 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187618324647936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2987860011008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4899 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2892300651008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46526443 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36794017 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚2322.034735382 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190510608259584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11142234 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0‹tÃ[#2—- _wandbruntime‚2322  _runtime‚2322jV8QÚ&˜ÔM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2892870010368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33749467 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22605009 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36801238 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184501879193600 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187655435612160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184501862154240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46535738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2882783040982 % _timestamp‚1.7843703809321797e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11144455 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187384662234582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46525747 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187655418572800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2892957954560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2988451205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83326983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33749232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46535738 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4899 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179592569267200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2882696903780 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187655418572800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2892957954560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36801238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176603945959424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36801236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187655435612160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83336976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83336976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36801236 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562­WZ˜$&™ÙM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187655418572800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2892957954560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36801238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176603945959424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36801236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187655435612160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83336976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83336976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36801236 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2892870010368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33749467 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22605009 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36801238 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184501879193600 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187655435612160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184501862154240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46535738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2882783040982 % _timestamp‚1.7843703809321797e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11144455 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187384662234582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46525747 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187655418572800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2892957954560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2988451205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83326983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33749232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46535738 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4899 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocÅè»ated‚179592569267200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2882696903780 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Œö7]2š- _wandbruntime‚2322  _runtime‚2322w¸ÌV›ÿ 0loss/SRqqt1/detection_loss‚0.836595394733761 /loss/FBt1/detection_loss‚0.9112376553316911 /loss/FBt2/detection_loss‚0.4095548154743339 %loss/total_loss‚8.591668331743639 trainer/global_step‚4899 _step‚397 1loss/FBt2/assignment_loss‚0.02538130784422159 % _timestamp‚1.7843703809339745e+09 _runtime‚2322.377574156 0loss/FRt1/assignment_loss‚1.7718833996202774 /loss/FRt1/detection_loss‚0.6055940008794358 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5717691111464852 2loss/SRqqt2/assignment_loss‚0.5603699158696053 /loss/FRt2/detection_loss‚1.5176070732187872 1loss/SRqqt2/detection_loss‚0.5972669428866625 0loss/FRt2/assignment_loss‚0.7439752447006365 0loss/FBt1/assignment_loss‚0.0404334700377414ʘã œ„ /loss/FRt1/detection_loss‚0.6055940008794358 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5717691111464852 2loss/SRqqt2/assignment_loss‚0.5603699158696053 /loss/FRt2/detection_loss‚1.5176070732187872 1loss/SRqqt2/detection_loss‚0.5972669428866625 0loss/FRt2/assignment_loss‚0.7439752447006365 0loss/FBt1/assignment_loss‚0.0404334700377414 0loss/SRqqt1/detection_loss‚0.836595394733761 /loss/FBt1/detection_loss‚0.9112376553316911 /loss/FBt2/detection_loss‚0.4095548154743339 %loss/total_loss‚8.591668331743639 trainer/global_step‚4899 _step‚397 1loss/FBt2/assignment_loss‚0.02538130784422159 % _timestamp‚1.7843703809339745e+09 _runtime‚2322.377574156 0loss/FRt1/assignment_loss‚1.7718833996202774·„®‘‚(:† Ï©íÒ“ð€ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚180.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚083„—Íž‚(: ϩíÒÒ†Ä disk.nvme0c0n1.in‚ 3219.59375% disk.nvme0n1.out‚4566.29443359375 disk.dm-0.in‚ 3219.59375' disk.nvme0c0n1.out‚4566.29443359375 cpu‚26.925046471910072 disk.nvme0n1.in‚ 3219.59375" disk.dm-0.out‚4566.29443359375 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2684.8984375 proc.cpu.threads‚17 disk.sda3.out‚ 58.23828125 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚5.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.109375& proc.memory.availableMB‚ 196.765625* proc.memory.percent‚16.387319564819336 network.sent‚ 6.478551e+06) disk./.usagePercent‚39.61449627711796 disk.rbd0.out‚0 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.sda.out‚ 58.23828125$ memory_percent‚98.79903793334961$ disk./.usageGB‚590.2846221923828 disk.dm-1.out‚ 58.23828125 network.recv‚762032 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0F|ÃÀœŸ‚(:‘ Þ©íÒ‹·‘Œ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R=whÉ ‚(:¾ Þ©íÒ¨¡»Ž" disk.dm-0.out‚4570.51611328125 disk.sda.out‚ 58.7578125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 3219.59375% disk.nvme0n1.out‚4570.51611328125 network.sent‚ 6.543129e+06 disk.rbd1.in‚0 disk.rbd1.out‚5.625 disk.dm-1.in‚0.109375 disk.sda1.out‚0 disk.sda3.in‚0.109375" proc.memory.rssMB‚ 2684.8984375 disk.nvme0c0n1.in‚ 3219.59375' disk.nvme0c0n1.out‚4570.51611328125 network.recv‚767124 disk.sda.in‚0.109375 disk.drbd2165.in‚0$ memory_percent‚98.79741668701172 cpu‚26.928292307123193% proc.memory.availableMB‚ 197.03125$ disk./.usageGB‚590.2847671508789 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 58.7578125* proc.memory.percent‚16.387319564819336 proc.cpu.threads‚17 disk.nvme0n1.in‚ 3219.59375) disk./.usagePercent‚39.61450600540426 disk.dm-1.out‚ 58.7578125úÕÿ 2¡- _wandbruntime‚2344  _runtime‚2344”¹¾Æ¢À _runtime‚2344.566607299 _step‚398 & lr-AdamW/pg1‚0.0008249031051586685 & lr-AdamW/pg2‚0.0008249031051586685 trainer/global_step‚4949 $ _timestamp‚1.784370403122737e+09„{‚†Ë£Å & lr-AdamW/pg1‚0.0008249031051586685 & lr-AdamW/pg2‚0.0008249031051586685 trainer/global_step‚4949 $ _timestamp‚1.784370403122737e+09 _runtime‚2344.566607299 _step‚398ŽfLTÖ2¤- _wandbruntime‚2344  _runtime‚2344Ìr¾Ð'¥ÊO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189199622623720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186289008924672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46992345 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11251533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37155065 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46982368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181332134701568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189199519447270 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37155067 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2920800876032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192394656695808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192394761672192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2910613699048 trainer/global_step‚4949 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178314585272320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192394761672192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37155065 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189473872859136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186288991885312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34076258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189473872859136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22824728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178314610176000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181331925105152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3017524525568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181331925105152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3017524525568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189199622623720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186289008924672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46992345 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11251533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37155065 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46982368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181332134701568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189199519447270 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37155067 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2920800876032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192394656695808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192394761672192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2910613699048 trainer/global_step‚4949 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178314585272320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192394761672192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37155065 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189473872859136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0;Íœ2§- _wandbruntime‚2344  _runtime‚2344béK|¨ÔM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46991649 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚186325463920640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2344.910616205 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47001640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843704034664402e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22829212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192432442002944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189236558804128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2911094883488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22829209 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3017928565248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37162288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37162288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚189510983823360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2911181020690 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47001640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2921458179584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84163928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181367622358528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178349521690624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37162286 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46991649 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34083110 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84163928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11253898 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2921370235392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84153935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192432442002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚189510966784000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚189510983823360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.ak W÷Õ llocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189236661980690 trainer/global_step‚4949 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84153935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34082872 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2921458179584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192NHßÔß&©ÙM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181367622358528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178349521690624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37162286 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46991649 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34083110 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84163928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11253898 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2921370235392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84153935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192432442002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚189510966784000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚189510983823360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189236661980690 trainer/global_step‚4949 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84153935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34082872 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2921458179584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46991649 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚186325463920640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2344.910616205 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47001640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843704034664402e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22829212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192432442002944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189236558804128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2911094883488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22829209 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3017928565248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37162288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37162288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚189510983823360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2911181020690 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47001640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2921458179584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84163928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327Ïâ2ª- _wandbruntime‚2344  _runtime‚2344³_ĉ«‚ /loss/FBt2/detection_loss‚0.3932505492668818 0loss/FRt1/assignment_loss‚1.9084092103772694 2loss/FBt2/assignment_loss‚0.055590881758345034 1loss/SRqqt2/detection_loss‚0.5929175827346864 2loss/SRqqt1/assignment_loss‚0.6833315056181339 epoch‚6 trainer/global_step‚4949 $ _timestamp‚1.784370403468718e+09 _runtime‚2344.912262424 %loss/total_loss‚8.728030102540599 0loss/FRt2/assignment_loss‚0.7678813335057851 2loss/SRqqt2/assignment_loss‚0.4531955150627344 /loss/FRt2/detection_loss‚1.5193649064976018 1loss/SRqqt1/detection_loss‚0.8817094527633793 /loss/FBt1/detection_loss‚0.9616503044962883 _step‚401 2loss/FBt1/assignment_loss‚0.011890293870344655 /loss/FRt1/detection_loss‚0.4988385665891487fJZ¦ ¬‡ %loss/total_loss‚8.728030102540599 0loss/FRt2/assignment_loss‚0.7678813335057851 2loss/SRqqt2/assignment_loss‚0.4531955150627344 /loss/FRt2/detection_loss‚1.5193649064976018 1loss/SRqqt1/detection_loss‚0.8817094527633793 /loss/FBt1/detection_loss‚0.9616503044962883 _step‚401 2loss/FBt1/assignment_loss‚0.011890293870344655 /loss/FRt1/detection_loss‚0.4988385665891487 /loss/FBt2/detection_loss‚0.3932505492668818 0loss/FRt1/assignment_loss‚1.9084092103772694 2loss/FBt2/assignment_loss‚0.055590881758345034 1loss/SRqqt2/detection_loss‚0.5929175827346864 2loss/SRqqt1/assignment_loss‚0.6833315056181339 epoch‚6 trainer/global_step‚4949 $ _timestamp‚1.784370403468718e+09 _runtime‚2344.912262424‘§’¸œ­‚(:‘ í©íÒøú‘‹ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚184.761' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.952200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m,„È®‚(:½ í©íÒðåÁŽ network.recv‚770750 disk.nvme0c0n1.in‚ 3323.5625 disk.rbd1.in‚0 disk.rbd1.out‚5.6875" proc.memory.rssMB‚ 2684.8984375* proc.memory.percent‚16.387319564819336$ disk./.usageGB‚590.2840805053711 disk.rbd0.out‚0 disk.sda3.in‚0.109375 disk.sda.out‚ 59.1484375 disk.drbd2165.in‚0 cpu‚26.931260701135415 disk.sda.in‚0.109375 network.sent‚ 6.559193e+06 disk.sda2.out‚0( proc.memory.availableMB‚ 196.11328125% disk.nvme0n1.out‚4575.05029296875 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚4575.05029296875 disk.rbd0.in‚0 disk.dm-0.in‚ 3323.5625 disk.sda3.out‚ 59.1484375' disk.nvme0c0n1.out‚4575.05029296875 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 59.1484375# memory_percent‚98.8030195236206 proc.cpu.threads‚17( disk./.usagePercent‚39.6144599240481 disk.nvme0n1.in‚ 3323.5625 disk.sda1.in‚0 disk.drbd2165.out‚0S"Ï2¯- _wandbruntime‚2366  _runtime‚2366¦4t£Å°¿ _step‚402 & lr-AdamW/pg1‚0.0008243467504065304 & lr-AdamW/pg2‚0.0008243467504065304 trainer/global_step‚4999 $ _timestamp‚1.784370425446644e+09 _runtime‚ 2366.89053419æïä/汀 & lr-AdamW/pg1‚0.0008243467504065304 & lr-AdamW/pg2‚0.0008243467504065304 trainer/global_step‚4999 $ _timestamp‚1.784370425446644e+09 _runtime‚ 2366.89053419 _step‚402’¬ ë)2²- _wandbruntime‚2366  _runtime‚2366´ ‡íÐ'³ÊO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194278705132032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191329421070336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191051622369828 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188112610691072 % _timestamp‚1.7843704254477804e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84974364 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11360339 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37516117 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180060185907200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚194278810108416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23048928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2939011678756 trainer/global_step‚4999 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2366.895653127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2949301101056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚194278705132032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191051519193378 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37516115 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188112593651712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11360568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191051622369828 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188112610691072 % _timestamp‚1.7843704254477804e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84974364 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11360339 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37516117 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180060185907200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚194278810108416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23048928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2939011678756 trainer/global_step‚4999 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2366.895653127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2949301101056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚194278705132032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191051519193378 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37516115 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188112593651712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11360568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194278705132032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488“‡Àá!2µ- _wandbruntime‚2367  _runtime‚2367½·ãÚ&¶ÔM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843704257881079e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11362774 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11362541 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚4999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚194316490439168 _runtime‚2367.232260963 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183142531319808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2939579000398 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚404 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37523338 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34415950 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3047458801664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37523336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34416186 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180095072518144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84980887 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191088558550236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191088661726798 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47467542 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191366514995200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47467542 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194316385455616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84990880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188149065687040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84990880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37523336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2949958404608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2949958404608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183142732520448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3047635098624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23053412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47457551 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180095097421824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚194316490439168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064$6ß&·ÙM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84980887 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191088558550236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191088661726798 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47467542 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191366514995200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47467542 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194316385455616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84990880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188149065687040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84990880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37523336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2949958404608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2949958404608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183142732520448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3047635098624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23053412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47457551 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180095097421824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚194316490439168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843704257881079e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11362774 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11362541 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚4999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚194316490439168 _runtime‚2367.232260963 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183142531319808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2939579000398 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚404 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37523338 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34415950 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3047458801664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37523336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34416186 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180095072518144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327”4­_2¸- _wandbruntime‚2367  _runtime‚2367pìýÊ ¹„ /loss/FRt1/detection_loss‚0.5824378037947469 1loss/SRqqt1/detection_loss‚0.8433917922080855 /loss/FBt2/detection_loss‚0.6018435298137261 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6544766185172319 4loss/FBt1/assignment_loss‚0.00029062518703426576 /loss/FRt2/detection_loss‚1.5213675027890574 %loss/total_loss‚8.677935690377204 % _timestamp‚1.7843704257900019e+09 2loss/SRqqt2/assignment_loss‚0.6275615625320161 2loss/FBt2/assignment_loss‚0.027280476421890437 1loss/SRqqt2/detection_loss‚0.6123401728998641 trainer/global_step‚4999 _runtime‚2367.233567871 /loss/FRt1/assignment_loss‚1.827639213297516 0loss/FRt2/assignment_loss‚0.8352113649958656 /loss/FBt1/detection_loss‚0.5440950279201692 _step‚405Pýã躉 4loss/FBt1/assignment_loss‚0.00029062518703426576 /loss/FRt2/detection_loss‚1.5213675027890574 %loss/total_loss‚8.677935690377204 % _timestamp‚1.7843704257900019e+09 2loss/SRqqt2/assignment_loss‚0.6275615625320161 2loss/FBt2/assignment_loss‚0.027280476421890437 1loss/SRqqt2/detection_loss‚0.6123401728998641 trainer/global_step‚4999 _runtime‚2367.233567871 /loss/FRt1/assignment_loss‚1.827639213297516 0loss/FRt2/assignment_loss‚0.8352113649958656 /loss/FBt1/detection_loss‚0.5440950279201692 _step‚405 /loss/FRt1/detection_loss‚0.5824378037947469 1loss/SRqqt1/detection_loss‚0.8433917922080855 /loss/FBt2/detection_loss‚0.6018435298137261 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6544766185172319•$‡Àœ»‚(:‘ ü©íÒØÀÓ‹ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉéÀ¼‚(:µ ü©íÒÕþ¥ disk.sda3.out‚ 59.546875) disk./.usagePercent‚39.61444481960357% disk.nvme0c0n1.out‚4577.943359375 disk.sda1.out‚0 disk.dm-1.out‚ 59.546875 network.sent‚ 6.60898e+06 disk.sda3.in‚0.109375 disk.rbd1.out‚5.6875 disk.sda2.out‚0$ disk./.usageGB‚590.2838554382324 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 3323.5625( proc.memory.availableMB‚ 196.43359375 proc.cpu.threads‚17 disk.sda.out‚ 59.546875 disk.nvme0c0n1.in‚ 3323.5625 disk.sda1.in‚0" proc.memory.rssMB‚ 2684.8984375* proc.memory.percent‚16.387319564819336 cpu‚26.934013098642072# disk.nvme0n1.out‚4577.943359375 disk.rbd0.in‚0 disk.dm-1.in‚0.109375$ memory_percent‚98.80106449127197 network.recv‚775687 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 3323.5625 disk.dm-0.out‚4577.943359375£¶ôꜽ‚(:‘ ‹ªíÒî‚ÞŠ gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚190.712' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.142399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛâµÆ¾‚(:» ‹ªíÒᨌ$ disk.nvme0n1.out‚4583.3408203125 cpu‚26.937036388273782& disk.nvme0c0n1.out‚4583.3408203125 disk.sda1.in‚0 disk.sda3.in‚0.109375$ memory_percent‚98.80328178405762 disk.rbd1.out‚5.8125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3323.5625" proc.memory.rssMB‚ 2684.8984375* proc.memory.percent‚16.387319564819336 proc.cpu.threads‚17$ disk./.usageGB‚590.2841415405273 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 59.7578125 disk.sda.out‚ 59.7578125 disk.drbd2165.out‚0 network.sent‚ 6.671768e+06 disk.nvme0c0n1.in‚ 3323.5625 disk.sda.in‚0.109375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 196.0703125 network.recv‚781038 disk.sda2.in‚0 disk.sda3.out‚ 59.7578125) disk./.usagePercent‚39.61446402016865 disk.nvme0n1.in‚ 3323.5625! disk.dm-0.out‚4583.3408203125 ²Ëö2¿- _wandbruntime‚2389  _runtime‚2389ÅçÑ3ÆÀÀ _step‚406 & lr-AdamW/pg1‚0.0008237839837748433 & lr-AdamW/pg2‚0.0008237839837748433 trainer/global_step‚5049 $ _timestamp‚1.784370447981705e+09 _runtime‚2389.425657009ÊŽ^ËÁÅ & lr-AdamW/pg1‚0.0008237839837748433 & lr-AdamW/pg2‚0.0008237839837748433 trainer/global_step‚5049 $ _timestamp‚1.784370447981705e+09 _runtime‚2389.425657009 _step‚406–Ó_Zõ2Â- _wandbruntime‚2389  _runtime‚2389ÕŠôUÃÊO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196162858544640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977801326080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196162753568256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2977801326080 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47914172 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37877167 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189936212457472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47924149 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196162858544640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2977889263104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37877165 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192903518939486 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚189936195418112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23273128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3076404719616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193184969281536 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196162753568256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47924149 trainer/global_step‚5049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M&¦5{@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34743272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181805736734720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192903622115936 _runtime‚2389.427279053 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11470388 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85791337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23273125 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2977889263104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193184969281536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34743516 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181805761638400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2967323521374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184882141454336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967409658464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193184952242176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3076576829440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47914172 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11470147¿&ª.Õ'ÄÏO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967409658464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193184952242176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3076576829440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47914172 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11470147 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196162858544640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977801326080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196162753568256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2977801326080 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47914172 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37877167 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189936212457472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47924149 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196162858544640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2977889263104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37877165 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192903518939486 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚189936195418112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23273128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3076404719616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193184969281536 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196162753568256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47924149 trainer/global_step‚5049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34743272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181805736734720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192903622115936 _runtime‚2389.427279053 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11470388 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85791337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23273125 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2977889263104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193184969281536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34743516 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181805761638400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2967323521374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184882141454336—(ž0ï2Å- _wandbruntime‚2389  _runtime‚2389>“©ëÚ&ÆÔM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196200433891840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2978370685440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181840673153024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181840648249344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193222080245760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196200538875392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85817832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚192940558296344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184917642795008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚408 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23277612 % _timestamp‚1.7843704483248162e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47923453 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34749872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11472487 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184917839801344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3076994545664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193222063206400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2978370685440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47923453 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11472263 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37884386 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189972684492800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192940661472906 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184917839801344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3076994545664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193222063206400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2978370685440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47923453 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11472263 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37884386 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189972684492800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192940661472906 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193222080245760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196200538875392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85817832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚192940558296344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184917642795008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚408 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23277612 % _timestamp‚1.7843704483248162e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47923453 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34749872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11472487 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195040517492736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35076948 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2995807638172 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2995721501082 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48390051 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198046906980864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183551337369600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198046906980864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3106091646464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191759797184512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38238215 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198046802004480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚183551312465920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3006301551104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚5099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195040500453376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86628268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3105927925248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3006301551104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38238217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23497328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23497325 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38238215 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11579384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚411 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3006389488128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186657429016064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38238217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48380074 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35076709 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198046802004480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48380074 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186657240391168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194755621862044 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194755518685594 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977ú4³Õ'ÒÏO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48390051 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11579620 _runtime‚2411.693803615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86618289 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195040517492736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35076948 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2995807638172 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2995721501082 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48390051 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198046906980864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183551337369600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198046906980864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3106091646464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191759797184512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38238215 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198046802004480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚183551312465920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3006301551104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚5099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195040500453376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86628268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3105927925248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3006301551104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38238217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23497328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23497325 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38238215 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11579384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚411 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3006389488128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186657429016064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38238217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48380074 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35076709 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198046802004480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48380074 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186657240391168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194755621862044 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194755518685594 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195040500453376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191759814223872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327›)÷aL2Ó- _wandbruntime‚2412  _runtime‚2412*Y5ãÌÔÔM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194792661219014 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195077611417600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38245436 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35083564 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183586223980544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195077628456960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23501809 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194792558042452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3006870910464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48399346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191796286259200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2996288822612 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48399346 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38245436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86644784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198084587311616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48389355 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38245438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195077611417600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11581523 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3106688453120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3106514253312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚412 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006958854656 XFDeviceStatsMoni”Ýï/tor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3006870910464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35083332 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006958854656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86634791 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186692937337344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2996374959814 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198084482328064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183586248884224mž6øß&ÕÙM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48389355 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38245438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195077611417600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11581523 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3106688453120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3106514253312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚412 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006958854656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3006870910464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35083332 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006958854656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86634791 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186692937337344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2996374959814 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198084482328064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183586248884224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194792661219014 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195077611417600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38245436 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35083564 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183586223980544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195077628456960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23501809 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194792558042452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3006870910464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48399346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191796286259200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2996288822612 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48399346 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38245436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86644784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198084587311616œµnÊ“2Ö- _wandbruntime‚2412  _runtime‚2412…= ×… _runtime‚2412.036587399 _step‚413 2loss/FBt2/assignment_loss‚0.018415708403257635 /loss/FBt2/detection_loss‚0.5304777692811239 0loss/FRt1/assignment_loss‚1.8349650876491774 4loss/FBt1/assignment_loss‚0.00035403761074744483 1loss/SRqqt1/detection_loss‚0.9102110042032309 % _timestamp‚1.7843704705930421e+09 %loss/total_loss‚9.693655133180695 epoch‚6 0loss/FRt2/assignment_loss‚0.7433087738972266 2loss/SRqqt1/assignment_loss‚0.6166301628027435 2loss/SRqqt2/assignment_loss‚0.5221654268345223 /loss/FRt1/detection_loss‚0.5445534610892796 1loss/SRqqt2/detection_loss‚0.5594959513484858 /loss/FBt1/detection_loss‚1.8607926045854886 /loss/FRt2/detection_loss‚1.5522851454754116 trainer/global_step‚5099ä/ UØŠ 0loss/FRt1/assignment_loss‚1.8349650876491774 4loss/FBt1/assignment_loss‚0.00035403761074744483 1loss/SRqqt1/detection_loss‚0.9102110042032309 % _timestamp‚1.7843704705930421e+09 %loss/total_loss‚9.693655133180695 epoch‚6 0loss/FRt2/assignment_loss‚0.7433087738972266 2loss/SRqqt1/assignment_loss‚0.6166301628027435 2loss/SRqqt2/assignment_loss‚0.5221654268345223 /loss/FRt1/detection_loss‚0.5445534610892796 1loss/SRqqt2/detection_loss‚0.5594959513484858 /loss/FBt1/detection_loss‚1.8607926045854886 /loss/FRt2/detection_loss‚1.5522851454754116 trainer/global_step‚5099 _runtime‚2412.036587399 _step‚413 2loss/FBt2/assignment_loss‚0.018415708403257635 /loss/FBt2/detection_loss‚0.5304777692811239ý¨ Ù‚(:„ ©ªíÒ¦“ÁŠ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚187.18' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,9zøÔÚ‚(:É ©ªíÒÓâ— disk.sda2.in‚0 disk.dm-0.in‚ 3340.09765625" disk.dm-0.out‚4591.60693359375 cpu‚26.942907100834017 disk.sda3.in‚0.109375 disk.dm-1.out‚ 60.3046875" proc.memory.rssMB‚ 2684.8984375 proc.cpu.threads‚17 disk.sda.in‚0.109375 disk.sda.out‚ 60.3046875 disk.rbd1.in‚0 network.recv‚789900# disk.nvme0c0n1.in‚ 3339.98828125% disk.nvme0n1.out‚4591.60693359375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375* proc.memory.percent‚16.387319564819336 disk.drbd2165.out‚0 disk.rbd1.out‚5.875 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚98.80061149597168 network.sent‚ 6.739535e+06$ disk./.usageGB‚590.2849159240723 disk.sda3.out‚ 60.3046875' disk.nvme0c0n1.out‚4591.60693359375! disk.nvme0n1.in‚ 3340.09765625) disk./.usagePercent‚39.61451598969809 disk.sda1.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 196.5078125¥¤©M‘Û‚(:† ¸ªíÒŒª¾Š gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚180.958' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öIßPÐÜ‚(:Å ¸ªíÒ½µ…Ž disk.sda.in‚0.109375 disk.sda.out‚ 60.6796875 disk.nvme0n1.in‚ 3403.4609375% disk.nvme0n1.out‚4594.44384765625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚5.875) disk./.usagePercent‚39.61451214958508 disk.sda3.out‚ 60.6796875 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.sda1.in‚0" proc.memory.rssMB‚ 2684.8984375" disk.dm-0.out‚4594.44384765625& proc.memory.availableMB‚ 196.515625" disk.nvme0c0n1.in‚ 3403.4609375 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 60.6796875$ memory_percent‚98.80056381225586 network.recv‚795131 disk.rbd0.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚4594.44384765625 network.sent‚ 6.802259e+06$ disk./.usageGB‚590.2848587036133 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.dm-0.in‚ 3403.4609375* proc.memory.percent‚16.387319564819336 cpu‚26.945575542247937À/Ã2Ý- _wandbruntime‚2434  _runtime‚2434ë'1çÇÞÁ _step‚414 & lr-AdamW/pg1‚0.0008226392510307776 & lr-AdamW/pg2‚0.0008226392510307776 trainer/global_step‚5149 % _timestamp‚1.7843704929076724e+09 _runtime‚2434.351623245ZùÔoÌßÆ _runtime‚2434.351623245 _step‚414 & lr-AdamW/pg1‚0.0008226392510307776 & lr-AdamW/pg2‚0.0008226392510307776 trainer/global_step‚5149 % _timestamp‚1.7843704929076724e+09ž„¹ûÏ2à- _wandbruntime‚2434  _runtime‚2434 ÇÃÐ'áÊO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38599267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199930850440704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199930955417088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199930850440704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185296888197120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3034889713152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193583415990272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3034889713152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11689321 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3024119480790 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38599267 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199930955417088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188432313704960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87445241 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38599265 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48855953 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38599265 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3135603909120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48845976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48855953 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196607518431702 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196896065703936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87455220 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚5149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196896048664576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196896065703936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196896048664576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35410849 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23721528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚196607621608152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188432517009920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199930850440704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199930955417088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199930850440704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185296888197120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3034889713152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193583415990272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3034889713152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11689321 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3024119480790 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38599267 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199930955417088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188432313704960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87445241 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38599265ŸðÎb…2ã- _wandbruntime‚2434  _runtime‚2434†¾1ÇÚ&äÔM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38606488 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48855257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185331824615424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35417503 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199968635747840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87471736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3035459079680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87471736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3035459079680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196644557788560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87461743 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843704932556973e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38606488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3136011960832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3024686802320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38606486 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185331799711744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48855257 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196933159628800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23726012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188467811672576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3024772939522 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196933176668160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199968530764288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5149 _step‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87461743 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48865248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188468014970368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3136190354944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3035371135488 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3035371135488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196644660965122 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193619870986240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48865248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35417280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196933176668160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11691491 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199968530764288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5149 _step‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87461743 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48865248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188468014970368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3136190354944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3035371135488 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3035371135488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196644660965122 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193619870986240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48865248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35417280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196933176668160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11691491 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38606488 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48855257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185331824615424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35417503 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199968635747840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87471736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3035459079680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87471736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3035459079680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196644557788560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87461743 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843704932556973e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38606488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3136011960832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3024686802320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38606486 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185331799711744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48855257 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196933159628800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23726012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188467811672576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3024772939522 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196933176668160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 lWÉZ2æ- _wandbruntime‚2434  _runtime‚2434°ã5‚ç 0loss/FRt2/assignment_loss‚0.8815145142100476 2loss/SRqqt1/assignment_loss‚0.6534095845665276 trainer/global_step‚5149 % _timestamp‚1.7843704932577686e+09 0loss/FRt1/assignment_loss‚1.7844128131866455 2loss/SRqqt2/assignment_loss‚0.5537509999851307 1loss/FBt1/assignment_loss‚0.11063553523984834 /loss/FRt1/detection_loss‚0.5836108165859785 1loss/SRqqt2/detection_loss‚0.6899634971158273 /loss/FBt1/detection_loss‚0.6221508172086694 _runtime‚2434.701307747 /loss/FRt2/detection_loss‚1.6078092076177537 1loss/SRqqt1/detection_loss‚0.8659563086756695 /loss/FBt2/detection_loss‚0.4332651853328571 $loss/total_loss‚8.79850034603656 epoch‚6 2loss/FBt2/assignment_loss‚0.012021066311605044 _step‚417E¡m è† /loss/FRt2/detection_loss‚1.6078092076177537 1loss/SRqqt1/detection_loss‚0.8659563086756695 /loss/FBt2/detection_loss‚0.4332651853328571 $loss/total_loss‚8.79850034603656 epoch‚6 2loss/FBt2/assignment_loss‚0.012021066311605044 _step‚417 0loss/FRt2/assignment_loss‚0.8815145142100476 2loss/SRqqt1/assignment_loss‚0.6534095845665276 trainer/global_step‚5149 % _timestamp‚1.7843704932577686e+09 0loss/FRt1/assignment_loss‚1.7844128131866455 2loss/SRqqt2/assignment_loss‚0.5537509999851307 1loss/FBt1/assignment_loss‚0.11063553523984834 /loss/FRt1/detection_loss‚0.5836108165859785 1loss/SRqqt2/detection_loss‚0.6899634971158273 /loss/FBt1/detection_loss‚0.6221508172086694 _runtime‚2434.701307747¡»ƒî¹œé‚(:‘ ǪíÒ÷óÒŠ gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚160.173' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.034600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tx„ûÌê‚(:Á ǪíÒ‘„‘" disk.nvme0c0n1.in‚ 3403.4609375$ memory_percent‚98.80049228668213 network.sent‚ 6.807136e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚60.84375 disk.rbd0.in‚0# disk./.usageGB‚590.285327911377 disk.sda2.in‚0 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 3403.4609375 proc.cpu.threads‚17( proc.memory.availableMB‚ 196.52734375 disk.dm-0.in‚ 3403.4609375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚4598.59326171875 disk.rbd0.out‚0) disk./.usagePercent‚39.61454363851179 network.recv‚798684 disk.sda3.out‚60.84375 disk.sda.out‚60.84375 disk.rbd1.out‚5.9375% disk.nvme0n1.out‚4598.59326171875* proc.memory.percent‚16.387319564819336 cpu‚26.948353176522467" proc.memory.rssMB‚ 2684.8984375 disk.dm-1.in‚0.109375" disk.dm-0.out‚4598.59326171875 disk.sda.in‚0.109375" 2ë- _wandbruntime‚2456  _runtime‚2456øêÇìÁ _runtime‚2456.859566024 _step‚418 & lr-AdamW/pg1‚0.0008220573032536172 & lr-AdamW/pg2‚0.0008220573032536172 trainer/global_step‚5199 % _timestamp‚1.7843705154156706e+09óª¹ÌíÆ & lr-AdamW/pg1‚0.0008220573032536172 & lr-AdamW/pg2‚0.0008220573032536172 trainer/global_step‚5199 % _timestamp‚1.7843705154156706e+09 _runtime‚2456.859566024 _step‚418¢’‰2î- _wandbruntime‚2456  _runtime‚2456‡îì ïÊO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195407017756672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49311878 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35744541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3063389938176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38960315 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190207582612480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_~à¨âÄ device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38960315 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198459518177810 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38960317 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3063302001152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3052517460498 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843705154167914e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3165093780480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3052603597588 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49311878 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3063389938176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11799024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201815003853312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190207391890432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187042488832000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88272193 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198751596875776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195407000717312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187042463928320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35744752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚2456.860842775 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49321855 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201814898876928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11798816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201815003853312 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚419 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3052603597588 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49311878 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3063389938176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11799024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201815003853312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190207391890432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187042488832000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88272193 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198751596875776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195407000717312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187042463928320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35744752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚2456.860842775 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49321855 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201814898876928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11798816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201815003853312 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚419 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5199£ÚôÛN2ñ- _wandbruntime‚2457  _runtime‚2457Œ¾EÚ&òÔM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚5199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49321159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201852684184064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38967536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49321159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23950212 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49331150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198788707840000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187077400346624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3063871360512 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201852579200512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88298688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88298688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 % _timestamp‚1.7843705157613935e+09 _step‚420 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38967536 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35751193 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190242893661184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195443472752640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88288695 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38967538 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195443489792000 _runtime‚2457.205634663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88288695 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49331150 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198788724879360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35751193 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190242893661184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195443472752640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88288695 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38967538 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195443489792000 _runtime‚2457.205634663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88288695 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49331150 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198788724879360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚5199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49321159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201852684184064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38967536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49321159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23950212 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49331150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198788707840000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187077400346624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3063871360512 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201852579200512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88298688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88298688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 % _timestamp‚1.7843705157613935e+09 _step‚420 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38967536 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203698947313152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49787757 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24169928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11908236 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191982440279552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚200311621100368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚2479.592820802 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39321367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191982637293056 $ _timestamp‚1.784370538148777e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3091890163200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197230619523072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24169925 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36078161 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188788064563200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3081001577296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3091890163200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200607162126336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49777780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197230602483712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚5249 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203698947313152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49787757 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24169928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11908236 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191982440279552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚200311621100368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚2479.592820802 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39321367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191982637293056 $ _timestamp‚1.784370538148777e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3091890163200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197230619523072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24169925 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36078161 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188788064563200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3081001577296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3091890163200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200607162126336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49777780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197230602483712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚5249 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36084909 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197267091558400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784370538498308e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3195163675136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39328588 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39328588 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11910500 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89125640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203736732620288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200644256051200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192017936455168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188822976077824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3092371585536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89125640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092459529728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092459529728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192018139752960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3081568898938 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200644273090560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49787061 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _runtime‚2479.942434347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36085134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24174409 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3081482761736 _step‚424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚200348557280776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203736627636736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11910722 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39328586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚5249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200644256051200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49787061 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200644273090560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200348660457338 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188822951174144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49797052 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24174409 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3081482761736 _step‚424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚200348557280776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203736627636736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11910722 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39328586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚5249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200644256051200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49787061 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200644273090560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200348660457338 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188822951174144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49797052 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36084909 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197267091558400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784370538498308e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3195163675136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39328588 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39328588 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11910500 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89125640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203736732620288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200644256051200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192017936455168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188822976077824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3092371585536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89125640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092459529728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092459529728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192018139752960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3081568898938 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473¨ô®ù2„- _wandbruntime‚2479  _runtime‚2479P@º‘ …ƒ 0loss/FRt1/assignment_loss‚1.8197571681368918 2loss/FBt1/assignment_loss‚0.026815900572055778 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚0.7325178224082086 2loss/SRqqt1/assignment_loss‚0.6143846232184973 1loss/SRqqt2/detection_loss‚0.6991196307680455 0loss/FBt2/detection_loss‚0.48063466461375354 %loss/total_loss‚8.888932031515193 2loss/SRqqt2/assignment_loss‚0.5069783317158424 2loss/FBt2/assignment_loss‚0.011931911060286779 /loss/FRt1/detection_loss‚0.6133998562159427 1loss/SRqqt1/detection_loss‚0.9845835010311162 % _timestamp‚1.7843705385001264e+09 .loss/FRt2/detection_loss‚1.548849459331227 /loss/FBt1/detection_loss‚0.8499591624433244 epoch‚6 _runtime‚2479.943667025 _step‚425²«sf†ˆ 0loss/FRt2/assignment_loss‚0.7325178224082086 2loss/SRqqt1/assignment_loss‚0.6143846232184973 1loss/SRqqt2/detection_loss‚0.6991196307680455 0loss/FBt2/detection_loss‚0.48063466461375354 %loss/total_loss‚8.888932031515193 2loss/SRqqt2/assignment_loss‚0.5069783317158424 2loss/FBt2/assignment_loss‚0.011931911060286779 /loss/FRt1/detection_loss‚0.6133998562159427 1loss/SRqqt1/detection_loss‚0.9845835010311162 % _timestamp‚1.7843705385001264e+09 .loss/FRt2/detection_loss‚1.548849459331227 /loss/FBt1/detection_loss‚0.8499591624433244 epoch‚6 _runtime‚2479.943667025 _step‚425 0loss/FRt1/assignment_loss‚1.8197571681368918 2loss/FBt1/assignment_loss‚0.026815900572055778 trainer/global_step‚5249©˜û¿›‡‚(: ôªí҇݊ gpu.0.gpu‚64.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.509999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]`cÀˆ‚(:· ôªíÒÕžÐ* disk./.usagePercent‚39.614554902843295$ disk./.usageGB‚590.2854957580566 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3456.046875 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375! disk.nvme0c0n1.in‚ 3456.046875" disk.dm-0.out‚4610.41064453125 disk.sda3.out‚61.5625 disk.rbd1.in‚0$ memory_percent‚98.80061149597168 disk.sda1.out‚0 disk.dm-0.in‚ 3456.046875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2684.8984375 proc.cpu.threads‚17 network.recv‚812587 disk.sda3.in‚0.109375 disk.sda.out‚61.5625' disk.nvme0c0n1.out‚4610.41064453125% disk.nvme0n1.out‚4610.41064453125 disk.sda2.out‚0' proc.memory.availableMB‚ 196.5078125* proc.memory.percent‚16.387319564819336 network.sent‚ 6.937335e+06 disk.rbd1.out‚6 disk.rbd0.out‚0 disk.dm-1.out‚61.5625 cpu‚26.956513882766405ž)®$2‰- _wandbruntime‚2501  _runtime‚2501†®àÂÇŠÁ & lr-AdamW/pg1‚0.0008208742914942666 & lr-AdamW/pg2‚0.0008208742914942666 trainer/global_step‚5299 % _timestamp‚1.7843705603525026e+09 _runtime‚2501.796286486 _step‚426%°clÌ‹Æ & lr-AdamW/pg1‚0.0008208742914942666 & lr-AdamW/pg2‚0.0008208742914942666 trainer/global_step‚5299 % _timestamp‚1.7843705603525026e+09 _runtime‚2501.796286486 _step‚426ª°û2Œ- _wandbruntime‚2501  _runtime‚2501U;EÐ'ÊO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3120390388224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202462710337536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190533615390720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50253659 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193757673502720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3109399557004 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205583100725760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89936076 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3120302451200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12018060 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205582995749376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36412188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89926097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50243682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199054204250112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3109313419914 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202462693298176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3120390388224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3120302451200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24394128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205583100725760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39682415 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202462710337536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843705603536458e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39682417 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146ŠÚˆ›•ŽÏO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12017842 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202163517670026 _runtime‚2501.797704503 _step‚427 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39682417 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202462710337536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190533615390720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50253659 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193757673502720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3109399557004 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205583100725760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89936076 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3120302451200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12018060 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205582995749376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36412188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89926097 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50243682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199054204250112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3109313419914 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202462693298176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3120390388224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3120302451200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24394128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205583100725760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39682415 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202462710337536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843705603536458e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39682417 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3120390388224 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36418617 _runtime‚2502.140489503 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89952592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12020225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199090676285440 _step‚428 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3120871810560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚5299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202499804262400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3224451092480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3109880741444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3120871810560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205620781056512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24398609 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39689638 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205620676072960 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89942599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3109966878646 % _timestamp‚1.7843705606963534e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202499821301760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89942599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24398612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202200660203446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3120959754752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50262954 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190568551809024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50262954 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205620781056512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89952592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12020008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3224627389440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36418837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39689638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202499804262400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39689636 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36418617 _runtime‚2502.140489503 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89952592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12020225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199090676285440 _step‚428 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3120871810560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚5299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202499804262400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3224451092480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3109880741444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3120871810560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205620781056512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24398609 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39689638 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205620676072960 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89942599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3109966878646 % _timestamp‚1.7843705606963534e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202499821301760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89942599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24398612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202200660203446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3120959754752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50262954 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190568551809024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50262954 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205620781056512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89952592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12020008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3224627389440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36418837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39689638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202499804262400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39689636 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50709584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12126696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204015620592584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40043467 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200877823055872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207467044185600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50719561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12126916 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195532557720064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40043465 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36745244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253366598144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207467044185600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204318258548736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0°x”>Õ'žÏO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207467044185600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50719561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12126916 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195532557720064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40043465 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36745244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253366598144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207467044185600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204318258548736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192279191121920 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207467149161984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _step‚431 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90753049 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204318258548736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24618325 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40043465 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50709584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12126696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204015620592584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40043467 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200877823055872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936¯d\1¤2Ÿ- _wandbruntime‚2524  _runtime‚2524?ûêªÙ& ÓM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204355369512960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3149372035584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204355352473600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200914295091200 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207504724509184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50718865 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207504829492736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3149459979776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40050688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192314127540224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204052659949554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 _runtime‚2524.183726854 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24622812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40050686 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36751932 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 $ _timestamp‚1.784370582739425e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207504829492736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204052556772992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3149372035584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204355352473600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚5349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90769551 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40050686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12129120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12128895 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40050688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195568258090496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3254130550272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36751704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192314102636544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200914278051840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90769551 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90779544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‹/^T“¡ØM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40050688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195568258090496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3254130550272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36751704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192314102636544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200914278051840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90769551 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90779544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204355369512960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3149372035584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204355352473600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200914295091200 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207504724509184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50718865 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207504829492736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3149459979776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40050688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192314127540224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204052659949554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 _runtime‚2524.183726854 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24622812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40050686 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36751932 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 $ _timestamp‚1.784370582739425e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207504829492736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204052556772992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3149372035584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204355352473600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚5349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90769551 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40050686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12129120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12128895° Ę̂2¢- _wandbruntime‚2524  _runtime‚2524Œó¾ £ÿ 0loss/SRqqt1/assignment_loss‚0.63573515241579 1loss/FBt1/assignment_loss‚0.01331279257586857 1loss/SRqqt1/detection_loss‚0.9072395713391854 1loss/SRqqt2/detection_loss‚0.7159304494844008 epoch‚6 1loss/SRqqt2/assignment_loss‚0.611261399707647 /loss/FBt2/detection_loss‚0.4886647248214347 %loss/total_loss‚9.005964704983423 trainer/global_step‚5349 % _timestamp‚1.7843705827408826e+09 _step‚433 2loss/FBt2/assignment_loss‚0.011344520964288377 /loss/FRt2/detection_loss‚1.5976268679022472 /loss/FBt1/detection_loss‚0.6993489247560501 _runtime‚2524.184536595 0loss/FRt2/assignment_loss‚0.9240714815702844 /loss/FRt1/detection_loss‚0.6118099985472251 0loss/FRt1/assignment_loss‚1.7896188208990007j½~ ¤„ 1loss/SRqqt2/assignment_loss‚0.611261399707647 /loss/FBt2/detection_loss‚0.4886647248214347 %loss/total_loss‚9.005964704983423 trainer/global_step‚5349 % _timestamp‚1.7843705827408826e+09 _step‚433 2loss/FBt2/assignment_loss‚0.011344520964288377 /loss/FRt2/detection_loss‚1.5976268679022472 /loss/FBt1/detection_loss‚0.6993489247560501 _runtime‚2524.184536595 0loss/FRt2/assignment_loss‚0.9240714815702844 /loss/FRt1/detection_loss‚0.6118099985472251 0loss/FRt1/assignment_loss‚1.7896188208990007 0loss/SRqqt1/assignment_loss‚0.63573515241579 1loss/FBt1/assignment_loss‚0.01331279257586857 1loss/SRqqt1/detection_loss‚0.9072395713391854 1loss/SRqqt2/detection_loss‚0.7159304494844008 epoch‚6±“® ¥‚(:… ¡«íÒ¡¦Ú‹ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÈô°Ì¦‚(:Á ¡«íÒîÍÜŒ disk.dm-0.in‚ 3456.046875 network.sent‚ 7.067664e+06 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.sda3.out‚ 62.0078125( proc.memory.availableMB‚ 196.49609375" proc.memory.rssMB‚ 2684.8984375 disk.sda2.in‚0 disk.drbd2165.out‚0$ memory_percent‚98.80068302154541 disk.nvme0n1.in‚ 3456.046875$ disk.nvme0n1.out‚4627.7607421875 network.recv‚826225 disk.sda1.out‚0 disk.rbd1.out‚6.1875$ disk./.usageGB‚590.2855262756348 disk.rbd0.out‚0! disk.dm-0.out‚4627.7607421875 disk.sda3.in‚0.109375 disk.rbd1.in‚0* proc.memory.percent‚16.387319564819336 cpu‚26.94480991757984 disk.dm-1.out‚ 62.0078125) disk./.usagePercent‚39.61455695090358 disk.rbd0.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 3456.046875 disk.sda.in‚0.109375 disk.sda.out‚ 62.0078125 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚4627.7607421875 proc.cpu.threads‚17,ú`Å2§- _wandbruntime‚2545  _runtime‚2545t¦ÇǨÁ % _timestamp‚1.7843706042870371e+09 _runtime‚2545.730842512 _step‚434 & lr-AdamW/pg1‚0.0008196658547924402 & lr-AdamW/pg2‚0.0008196658547924402 trainer/global_step‚5399TK÷×Ì©Æ % _timestamp‚1.7843706042870371e+09 _runtime‚2545.730842512 _step‚434 & lr-AdamW/pg1‚0.0008196658547924402 & lr-AdamW/pg2‚0.0008196658547924402 trainer/global_step‚5399²®,»2ª- _wandbruntime‚2545  _runtime‚2545 ãOÐ'«ÊO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40404517 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24842525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3166109379330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3177302901248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3177390838272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91589980 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _runtime‚2545.735671562 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202701407782912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202701424822272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40404515 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51175486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194024791756800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209351197598208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197307796789760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205867517162242 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209351092621824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206173789720576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51175486 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197307591387648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91580001 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206173789720576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40404515 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3282824534528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194024766853120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206173806759936 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51185463 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40404517 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3177302901248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚5399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3283005032960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205867620338692 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843706042881007e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3177390838272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209351092621824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206173806759936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24842528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979A“ópý%¬ÏO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51185463 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205867620338692 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843706042881007e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3177390838272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209351092621824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206173806759936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24842528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40404517 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24842525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3166109379330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3177302901248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3177390838272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91589980 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _runtime‚2545.735671562 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202701407782912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202701424822272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40404515 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51175486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194024791756800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209351197598208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197307796789760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205867517162242 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209351092621824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206173789720576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51175486 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197307591387648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91580001 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206173789720576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40404515 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3282824534528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194024766853120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206173806759936 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51185463 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40404517 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3177302901248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚5399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3283005032960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3166676700860 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40411736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91606496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40411738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚5399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209388772945408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91596503 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40411738 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209388877928960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197343093219840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194059703271424 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37085397 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3177872260608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40411736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202737896857600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3283414852096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91596503 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3177960204800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194059678367744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3177960204800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51194758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202737879818240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209388877928960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205904659695662 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3177872260608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2546.076659924 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51184767 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205904556519100 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51194758 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206210917724160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206210900684800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12238616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197343294420480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91606496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206210900684800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843706046325028e+09 _step‚436p–Àß&¯ÙM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209388877928960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205904659695662 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3177872260608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2546.076659924 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51184767 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205904556519100 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51194758 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206210917724160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206210900684800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12238616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197343294420480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91606496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206210900684800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843706046325028e+09 _step‚436 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24847012 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3166676700860 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40411736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91606496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40411738 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚5399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209388772945408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91596503 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40411738 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209388877928960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197343093219840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194059703271424 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37085397 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3177872260608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40411736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202737896857600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3283414852096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91596503 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3177960204800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194059678367744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3177960204800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51194758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202737879818240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0´Úûq2°- _wandbruntime‚2546  _runtime‚2546g_±‚ trainer/global_step‚5399 2loss/FBt2/assignment_loss‚0.000708725348083799 1loss/SRqqt1/detection_loss‚0.9158326604890383 % _timestamp‚1.7843706046343358e+09 0loss/FRt2/assignment_loss‚0.9260401353240013 2loss/SRqqt2/assignment_loss‚0.5056996822856686 /loss/FRt2/detection_loss‚1.5543184814784952 /loss/FBt2/detection_loss‚0.5776084968173463 2loss/FBt1/assignment_loss‚0.001114546215413939 /loss/FRt1/detection_loss‚0.5830382751385907 1loss/SRqqt2/detection_loss‚0.5921167050918588 /loss/FBt1/detection_loss‚0.6350267126279718 epoch‚6 _runtime‚2546.077909323 _step‚437 /loss/FRt1/assignment_loss‚1.739784571661878 2loss/SRqqt1/assignment_loss‚0.7695287246321889 %loss/total_loss‚8.800817717110537Iyg ²‡ 2loss/FBt1/assignment_loss‚0.001114546215413939 /loss/FRt1/detection_loss‚0.5830382751385907 1loss/SRqqt2/detection_loss‚0.5921167050918588 /loss/FBt1/detection_loss‚0.6350267126279718 epoch‚6 _runtime‚2546.077909323 _step‚437 /loss/FRt1/assignment_loss‚1.739784571661878 2loss/SRqqt1/assignment_loss‚0.7695287246321889 %loss/total_loss‚8.800817717110537 trainer/global_step‚5399 2loss/FBt2/assignment_loss‚0.000708725348083799 1loss/SRqqt1/detection_loss‚0.9158326604890383 % _timestamp‚1.7843706046343358e+09 0loss/FRt2/assignment_loss‚0.9260401353240013 2loss/SRqqt2/assignment_loss‚0.5056996822856686 /loss/FRt2/detection_loss‚1.5543184814784952 /loss/FBt2/detection_loss‚0.5776084968173463µPÚù›³‚(: °«íÒýÁŠ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úã’“Ç´‚(:¼ °«íÒ“Þ”Œ disk.nvme0c0n1.in‚3505.875 disk.sda2.in‚0% disk.nvme0n1.out‚4633.35400390625 cpu‚26.940682027891697$ disk./.usageGB‚590.2861061096191 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.387319564819336 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚3505.875 disk.sda.out‚ 62.40234375 network.sent‚ 7.12956e+06 disk.sda2.out‚0 disk.dm-1.out‚ 62.40234375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚4633.35400390625' proc.memory.availableMB‚ 131.9609375 network.recv‚830355 disk.dm-0.in‚3505.875 disk.sda3.in‚0.109375 disk.sda3.out‚ 62.40234375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2684.8984375) disk./.usagePercent‚39.61459586404877" disk.dm-0.out‚4633.35400390625 disk.rbd1.out‚6.1875 disk.rbd0.in‚0# memory_percent‚99.1945743560791 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375Îìí1›µ‚(: ¿«íÒðêÿ‰ gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.724000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍØ·Ò¶‚(:Ç ¿«íÒë·ÅŒ disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚6.25$ disk.nvme0n1.out‚4638.5771484375 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 3505.87890625! disk.dm-0.out‚4638.5771484375$ memory_percent‚99.19576644897461 network.recv‚835881 disk.sda.out‚ 62.58984375! disk.nvme0n1.in‚ 3505.87890625* proc.memory.percent‚16.387319564819336 proc.cpu.threads‚17 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 3505.87890625 disk.sda.in‚0.109375 network.sent‚ 7.192133e+06& disk.nvme0c0n1.out‚4638.5771484375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚26.936581690286744 disk.sda3.out‚ 62.58984375& proc.memory.availableMB‚ 131.765625" proc.memory.rssMB‚ 2684.8984375 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 62.58984375) disk./.usagePercent‚39.61473897226042$ disk./.usageGB‚590.2882385253906 disk.sda3.in‚0.109375 disk.rbd1.in‚0Rí¯2·- _wandbruntime‚2567  _runtime‚2567 Ž«EǸÁ _runtime‚2567.764749371 _step‚438 & lr-AdamW/pg1‚0.0008190521262189652 & lr-AdamW/pg2‚0.0008190521262189652 trainer/global_step‚5449 % _timestamp‚1.7843706263208957e+09ôÐ5KÌ¹Æ & lr-AdamW/pg1‚0.0008190521262189652 & lr-AdamW/pg2‚0.0008190521262189652 trainer/global_step‚5449 % _timestamp‚1.7843706263208957e+09 _runtime‚2567.764749371 _step‚438¶Œ¦¡Ñ2º- _wandbruntime‚2567  _runtime‚2567c˜õÀ»ÊO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37411898 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3205891063296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208029337931776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207719620084800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211235141058048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3205803126272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚199082867396096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40765565 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208029354971136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51651365 TGDeviceStatsMon¼¬] itor.on_train_batch_start/allocation.small_pool.allocated‚51651365 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12344945 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚2567.769680439 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51641388 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37411670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3194507359038 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204525026588672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3205891063296 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 % _timestamp‚1.7843706263219633e+09 _step‚439 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211235246034432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199082674576896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3205891063296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208029337931776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207719620084800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211235141058048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3205803126272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚199082867396096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40765565 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208029354971136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51651365 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51651365 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12344945 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚2567.769680439 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51641388 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37411670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3194507359038 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204525026588672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3205891063296 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 % _timestamp‚1.7843706263219633e+09 _step‚439 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211235246034432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199082674576896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3195160817770 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3206372485632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92433448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37418515 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25071212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12347303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚199118368983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195805254098944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40772788 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51650669 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40772786 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199118176171520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211272821381632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚207756556265208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51650669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12347078 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37418287 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92423455 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51660660 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3206460429824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚440 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211272926365184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92423455 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3312922072576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3195074680568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚207756659441770 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 _runtime‚2568.104839473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÛKqeí%¿ØM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208066465935360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3206372485632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37418287 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92423455 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51660660 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3206460429824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚440 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211272926365184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92423455 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3312922072576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3195074680568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚207756659441770 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 _runtime‚2568.104839473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784370626660711e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40772788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3313089980928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3195160817770 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3206372485632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92433448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37418515 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25071212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12347303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚199118368983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195805254098944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40772788 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51650669 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40772786 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199118176171520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211272821381632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚207756556265208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51650669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12347078 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3234391288320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41126615 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213119189494272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341833186304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213119294470656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41126615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3234303351296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚5499 $ _timestamp‚1.784370648158404e+09ͱpyÔ'ÊÎO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206348611315712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25290928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200857952709632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206348628355072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3222905338746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209571516654458 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52107290 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209884886142976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52117267 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209884886142976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197515918315520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197515943219200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213119189494272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341833186304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213119294470656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41126615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3234303351296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚5499 $ _timestamp‚1.784370648158404e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3234391288320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200857751501824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3234391288320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41126615 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843706484994338e+09 _step‚444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93260400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213156974801408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41133836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25295409 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41133838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209608659187878 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213156974801408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209921997107200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93250407 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52126562 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37751538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12456365 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3223472660276 trainer/global_step‚5499 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200893250585600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41133838 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52116571 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213156869817856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12456129 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚2589.943577083 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93260400 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3234960654848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3234872710656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342597052416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209608556011316 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93250407 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41133836 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209921997107200 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843706484994338e+09 _step‚444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93260400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213156974801408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41133836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25295409 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41133838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209608659187878 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213156974801408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209921997107200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93250407 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52126562 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37751538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12456365 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3223472660276 trainer/global_step‚5499 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200893250585600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41133838 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52116571 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213156869817856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12456129 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚2589.943577083 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93260400 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3234960654848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3234872710656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342597052416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209608556011316 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93250407 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41133836 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209921997107200 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215003237930496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41487667 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3251303318454 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211740434354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3251389455544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3262803576320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3262803576320 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215003237930496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41487667 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3251303318454 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211740434354176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3251389455544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3262803576320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¿kãý2Û- _wandbruntime‚2612  _runtime‚2612Ѹ‰:Ù&ÜÓM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211777545318400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211777562357760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38086137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25519612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199296405561344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚211460658933986 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52592464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _runtime‚2612.640227538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94087352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25519609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202668531920384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚208208702156800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3371933547008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚5549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41494886 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3263460879872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38085902 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3372101455360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215041023237632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52592464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215041023237632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211460555757424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3251956777186 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41494886 $ _timestamp‚1.784370671196067e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94077359 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚208208685117440 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3263372935680 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12566293 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3263460879872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52582473 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199296430465024 _step‚448 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94087352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3263372935680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215040918254080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94077359 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202668339108352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128z’'Þ&ÝØM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41494886 $ _timestamp‚1.784370671196067e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94077359 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚208208685117440 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3263372935680 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12566293 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3263460879872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52582473 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199296430465024 _step‚448 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94087352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3263372935680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215040918254080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94077359 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202668339108352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211777545318400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211777562357760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38086137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25519612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199296405561344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚211460658933986 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52592464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _runtime‚2612.640227538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94087352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25519609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202668531920384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚208208702156800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3371933547008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚5549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41494886 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3263460879872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38085902 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3372101455360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215041023237632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52592464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215041023237632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211460555757424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3251956777186 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192À™òH"2Þ- _wandbruntime‚2612  _runtime‚2612w)[ ß -loss/FBt1/detection_loss‚0.84365409668535 %loss/total_loss‚8.910931960358404 /loss/FRt1/detection_loss‚0.5739342869018133 1loss/SRqqt1/detection_loss‚0.7583845872982775 0loss/SRqqt2/detection_loss‚0.796675941731866 /loss/FBt2/detection_loss‚0.5592522488128561 _runtime‚2612.641724494 _step‚449 0loss/FRt2/assignment_loss‚0.8469759216722177 2loss/SRqqt1/assignment_loss‚0.5817172770940459 2loss/SRqqt2/assignment_loss‚0.5275260041132177 2loss/FBt1/assignment_loss‚0.012260878198213376 3loss/FBt2/assignment_loss‚0.0016007348721134196 trainer/global_step‚5549 % _timestamp‚1.7843706711981723e+09 epoch‚6 0loss/FRt1/assignment_loss‚1.8038492886711368 /loss/FRt2/detection_loss‚1.6051006943072972ymíç à† /loss/FRt1/detection_loss‚0.5739342869018133 1loss/SRqqt1/detection_loss‚0.7583845872982775 0loss/SRqqt2/detection_loss‚0.796675941731866 /loss/FBt2/detection_loss‚0.5592522488128561 _runtime‚2612.641724494 _step‚449 0loss/FRt2/assignment_loss‚0.8469759216722177 2loss/SRqqt1/assignment_loss‚0.5817172770940459 2loss/SRqqt2/assignment_loss‚0.5275260041132177 2loss/FBt1/assignment_loss‚0.012260878198213376 3loss/FBt2/assignment_loss‚0.0016007348721134196 trainer/global_step‚5549 % _timestamp‚1.7843706711981723e+09 epoch‚6 0loss/FRt1/assignment_loss‚1.8038492886711368 /loss/FRt2/detection_loss‚1.6051006943072972 -loss/FBt1/detection_loss‚0.84365409668535 %loss/total_loss‚8.910931960358404ÁÚK&Áá‚(:¶ û«íÒð¿ó disk.dm-1.out‚ 63.8046875" proc.memory.rssMB‚ 2684.8984375 network.recv‚853736$ disk.nvme0n1.out‚4661.1435546875( proc.memory.availableMB‚ 131.96484375& disk.nvme0c0n1.out‚4661.1435546875 disk.dm-1.in‚0.109375 disk.sda.out‚ 63.8046875 network.sent‚ 7.32768e+06( disk./.usagePercent‚39.6148098863474! disk.dm-0.out‚4661.1435546875 disk.rbd1.out‚6.375 disk.sda1.in‚0 disk.sda3.out‚ 63.8046875 disk.sda.in‚0.109375 proc.cpu.threads‚17 disk.sda1.out‚0 cpu‚26.937044735456276 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚3562.625 disk.sda3.in‚0.109375 disk.nvme0c0n1.in‚3562.625* proc.memory.percent‚16.387319564819336 disk.sda2.in‚0$ memory_percent‚99.19455051422119 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚590.2892951965332 disk.nvme0n1.in‚3562.625 disk.rbd0.in‚0 disk.sda2.out‚0Šç ‘‘â‚(:† û«íÒИȌ gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚179.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â?•2ã- _wandbruntime‚2634  _runtime‚2634V Ê'ÇäÁ & lr-AdamW/pg1‚0.0008171730178731473 & lr-AdamW/pg2‚0.0008171730178731473 trainer/global_step‚5599 % _timestamp‚1.7843706935267112e+09 _runtime‚2634.971718906 _step‚450Kq²ÉÌåÆ _step‚450 & lr-AdamW/pg1‚0.0008171730178731473 & lr-AdamW/pg2‚0.0008171730178731473 trainer/global_step‚5599 % _timestamp‚1.7843706935267112e+09 _runtime‚2634.971718906Â^¦¬J2æ- _wandbruntime‚2634  _runtime‚2634Éú”Ï•çÉO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3291303801344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12673905 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204407889500160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3291391738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41848717 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3401002317824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3279701298162 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213595982565376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216887286366720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201007094681600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 20749926"šç:4 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213275516146674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25739328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201007069777920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38413230 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213595982565376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3291303801344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53049071 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41848717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53039094 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41848715 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209995814848512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216887391343104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3291391738368"åiÓÔ'èÎO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12673905 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204407889500160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3291391738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41848717 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3401002317824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3279701298162 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213595982565376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216887286366720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201007094681600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213275516146674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25739328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201007069777920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38413230 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213595982565376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3291303801344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53049071 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41848717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53039094 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41848715 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209995814848512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216887391343104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3291391738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3291303801344ÃkAã2é- _wandbruntime‚2635  _runtime‚2635˹ð\Ú&êÔM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 % _timestamp‚1.7843706938717206e+09 _step‚452 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25743809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204443590423552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38420158 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚210032303923200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12676114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41855938 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3291961104896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204443389222912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3280268619692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213633110568960 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38419923 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213312658680094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53058366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25743812 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94904311 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53048375 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3291873160704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213633093529600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53048375 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3401407930368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94914304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94914304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213633093529600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201041981292544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216924966690304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3280354756894 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213633110568960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41855936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12676346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210032286883840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216925071673856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41855936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5599 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3401584227328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213312555503532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216925071673856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94904311 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38419923 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213312658680094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53058366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25743812 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94904311 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53048375 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3291873160704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213633093529600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53048375 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3401407930368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94914304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94914304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213633093529600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201041981292544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216924966690304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3280354756894 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213633110568960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41855936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12676346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210032286883840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216925071673856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocatedð[ ž _bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41855936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5599 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3401584227328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213312555503532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216925071673856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94904311 B&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5097814798355103 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_4‚0.6580485105514526 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6914175748825073 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06290670484304428 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9962481260299683 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7200372219085693 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6944444179534912 epoch‚6 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12846733629703522 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013224276714026928 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 +particle/specificity‚0.9059596061706543 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3439269959926605 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.642745852470398 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7167418599128723 'particle/f_score‚0.6413049101829529 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9628495573997498 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9952699542045593 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.604708731174469 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13747118413448334 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8811043500900269 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9013649225234985 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5226880311965942 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9961739778518677 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1516176164150238 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7324693202972412 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988896250724792 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6402567028999329 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003968254197388887 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06759780645370483 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6779286861419678 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023299758322536945 +jet/accuracy_1_of_1‚0.28121617436408997 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5879479646682739 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06902456283569336 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8482455015182495 % _timestamp‚1.7843707162395737e+09 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.997103214263916 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9236235618591309 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10288155823945999 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12256143242120743 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02399660088121891 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14056549966335297 .particle/accuracy_3_of_3‚0.999514102935791 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.602118194103241 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6535277366638184 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.45098957419395447 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8499999642372131 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚5606 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15501734614372253 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6245946288108826 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6386505961418152 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5535348057746887 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1393631249666214 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11121898889541626 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20156250894069672 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9448983073234558 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8952260613441467 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08993659913539886 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.63910973072052 6validation_average_jet_accuracy‚0.3270350992679596 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 particle/accuracy_5_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12256143242120743 _step‚454 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5300409197807312 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988964796066284 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12674042582511902 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.776495635509491 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4305555820465088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982557892799377 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27799251675605774 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913784861564636 ,jet/accuracy_4_of_4‚0.016514042392373085 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8535371422767639 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5882812738418579 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9902577996253967 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 /particle/accuracy_4_of_4‚0.9407414197921753 'particle/accuracy‚0.847632884979248 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9899895787239075 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13778053224086761 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.738598108291626 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11745592206716537 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08349189907312393 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06758235394954681 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48526623845100403 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19813333451747894 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8482455015182495 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2657.684088735 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6305664777755737 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6539185643196106 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06880161911249161 *jet/accuracy_1_of_2‚0.7406067848205566 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21773511171340942 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6067360639572144 +jet/accuracy_3_of_4‚0.09838317334651947 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6344799399375916 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7220399975776672 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5669430494308472 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5370867848396301 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 (loss/val_total_loss‚8.47708491711165 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1273788958787918 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8595542907714844 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6426213979721069 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5827121138572693 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1579589545726776 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968549013137817 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06290670484304428 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.259312242269516 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6120893955230713 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11205512285232544 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6255722641944885 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6356521844863892 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15366478264331818 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7324693202972412 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39772728085517883 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10577229410409927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6089445352554321 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5973680019378662 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09254279732704163 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5848342776298523 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5859255194664001 +jet/accuracy_2_of_2‚0.13066552579402924 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982757031917572 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016772547736763954 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11032871156930923 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6465960144996643 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5049999952316284 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968941807746887 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5912020802497864 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_4‚0.9970238208770752 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6089445352554321 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.354074627161026 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6810548305511475 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.721808671951294 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6343730092048645 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12312255799770355 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611110746860504 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12684357166290283 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_3‚0.07065365463495255 jet/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961903691291809 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028025159612298012 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3525035083293915 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11813700944185257 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6948155164718628 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6033479571342468 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8701127767562866 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0”`éŸ'õÌ{ 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3525035083293915 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11813700944185257 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6948155164718628 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6033479571342468 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8701127767562866 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.355197012424469 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42102763056755066 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5097814798355103 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_4‚0.6580485105514526 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6914175748825073 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06290670484304428 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9962481260299683 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7200372219085693 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6944444179534912 epoch‚6 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12846733629703522 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013224276714026928 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 +particle/specificity‚0.9059596061706543 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3439269959926605 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.642745852470398 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7167418599128723 'particle/f_score‚0.6413049101829529 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9628495573997498 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9952699542045593 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.604708731174469 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13747118413448334 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8811043500900269 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9013649225234985 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5226880311965942 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9961739778518677 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1516176164150238 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7324693202972412 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988896250724792 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6402567028999329 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003968254197388887 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06759780645370483 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6779286861419678 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0023299758322536945 +jet/accuracy_1_of_1‚0.28121617436408997 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5879479646682739 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06902456283569336 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8482455015182495 % _timestamp‚1.7843707162395737e+09 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.997103214263916 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9236235618591309 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10288155823945999 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12256143242120743 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02399660088121891 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14056549966335297 .particle/accuracy_3_of_3‚0.999514102935791 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.602118194103241 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6535277366638184 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.45098957419395447 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8499999642372131 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚5606 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15501734614372253 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6245946288108826 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6386505961418152 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5535348057746887 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1393631249666214 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11121898889541626 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20156250894069672 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9448983073234558 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8952260613441467 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08993659913539886 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.63910973072052 6validation_average_jet_accuracy‚0.3270350992679596 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 particle/accuracy_5_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12256143242120743 _step‚454 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5300409197807312 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988964796066284 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12674042582511902 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.776495635509491 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4305555820465088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982557892799377 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27799251675605774 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913784861564636 ,jet/accuracy_4_of_4‚0.016514042392373085 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8535371422767639 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5882812738418579 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9902577996253967 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 /particle/accuracy_4_of_4‚0.9407414197921753 'particle/accuracy‚0.847632884979248 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9899895787239075 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13778053224086761 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.738598108291626 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11745592206716537 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08349189907312393 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06758235394954681 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48526623845100403 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19813333451747894 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8482455015182495 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚2657.684088735 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6305664777755737 7 Purity/*FRt0SRqqÁ9õ3t1FBt/FRt_purity‚0.6539185643196106 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06880161911249161 *jet/accuracy_1_of_2‚0.7406067848205566 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21773511171340942 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6067360639572144 +jet/accuracy_3_of_4‚0.09838317334651947 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6344799399375916 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7220399975776672 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5669430494308472 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5370867848396301 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 (loss/val_total_loss‚8.47708491711165 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1273788958787918 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8595542907714844 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6426213979721069 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5827121138572693 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1579589545726776 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968549013137817 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06290670484304428 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.259312242269516 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6120893955230713 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11205512285232544 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6255722641944885 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6356521844863892 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15366478264331818 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7324693202972412 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39772728085517883 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10577229410409927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6089445352554321 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5973680019378662 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09254279732704163 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5848342776298523 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5859255194664001 +jet/accuracy_2_of_2‚0.13066552579402924 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982757031917572 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016772547736763954 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11032871156930923 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6465960144996643 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5049999952316284 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968941807746887 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5912020802497864 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_4‚0.9970238208770752 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6089445352554321 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.354074627161026 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6810548305511475 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.721808671951294 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6343730092048645 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12312255799770355 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611110746860504 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12684357166290283 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_3‚0.07065365463495255 jet/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961903691291809 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028025159612298012 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0Æîf‘ö‚(:† ¨¬íÒבՊ gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚123.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ŽŒ'·÷‚(:¬ ¨¬íÒÙ•û disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚3607.25 disk.dm-1.in‚0.109375 disk.sda3.out‚64.625$ memory_percent‚98.80533218383789* proc.memory.percent‚16.413283348083496 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0.109375% disk.nvme0n1.out‚4740.98779296875 proc.cpu.threads‚17* disk./.usagePercent‚39.614833951055616 disk.rbd0.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚4740.98779296875# proc.memory.rssMB‚ 2689.15234375 cpu‚26.95913001929237 network.recv‚868273 disk.rbd0.in‚0 disk.dm-0.in‚3607.25 disk.rbd1.in‚0 network.sent‚ 7.458307e+06" disk.dm-0.out‚4740.98779296875 disk.nvme0c0n1.in‚3607.25& proc.memory.availableMB‚ 195.734375 disk.sda3.in‚0.109375 disk.sda.out‚64.625$ disk./.usageGB‚590.2896537780762 disk.dm-1.out‚64.625 disk.drbd2165.out‚0 disk.rbd1.out‚6.4375$ó¦2ø- _wandbruntime‚2677  _runtime‚2677¶SÇùÁ & lr-AdamW/pg1‚0.0008165340409913553 & lr-AdamW/pg2‚0.0008165340409913553 trainer/global_step‚5649 % _timestamp‚1.7843707361884696e+09 _runtime‚2677.632309341 _step‚455ȸ¦vÌúÆ trainer/global_step‚5649 % _timestamp‚1.7843707361884696e+09 _runtime‚2677.632309341 _step‚455 & lr-AdamW/pg1‚0.0008165340409913553 & lr-AdamW/pg2‚0.0008165340409913553ÇnSjì2û- _wandbruntime‚2677  _runtime‚2677çtÏYÐ'üÊO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚216412510130176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212776634488832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42444293 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212776651528192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚216412510130176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26068105 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207114084582400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26068108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203667183837184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53658551 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38917307 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3446925648896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3324587879316 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42444295 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53648574 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3324674016406 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843707361898549e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3336405820928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38917076 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53658551 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96102846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219748932990464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12848971 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219748828014080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216101222368148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2677.633870227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚216412527169536/eÈÊÕ'ýÏO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3336405820928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38917076 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53658551 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96102846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219748932990464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12848971 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219748828014080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216101222368148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2677.633870227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚216412527169536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚216412510130176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212776634488832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42444293 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212776651528192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚216412510130176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26068105 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207114084582400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26068108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203667183837184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53658551 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38917307 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3446925648896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3324587879316 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42444295 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53648574 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3324674016406 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843707361898549e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979ÈòÊÁ32þ- _wandbruntime‚2677  _runtime‚2677«¿jÿÔM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2677.977549235 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53667846 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42451514 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3325241338048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚215 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3447511325696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚216449638133760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 trainer/global_step‚5649 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53657855 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3325155200846 % _timestamp‚1.7843707365333989e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219786508337664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12851275 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203702095351808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3447685525504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216138261725006 _step‚457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336975187456 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53657855 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚216449638133760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207149581773824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96109369 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚216449621094400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53667846 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96119362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216138364901568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42451516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203702070448128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3336887243264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚219786613321216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3336887243264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38923649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336975187456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101\ V±ß&€ÙM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3447685525504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216138261725006 _step‚457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336975187456 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53657855 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚216449638133760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207149581773824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96109369 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚216449621094400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53667846 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96119362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216138364901568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42451516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203702070448128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3336887243264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚219786613321216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3336887243264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38923649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336975187456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2677.977549235 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53667846 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42451514 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3325241338048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚215 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3447511325696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚216449638133760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 trainer/global_step‚5649 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53657855 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3325155200846 % _timestamp‚1.7843707365333989e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219786508337664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12851275 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203702095351808 Bõ_LJÁ & lr-AdamW/pg1‚0.0008158887773664137 & lr-AdamW/pg2‚0.0008158887773664137 trainer/global_step‚5699 % _timestamp‚1.7843707579497998e+09 _runtime‚2699.393617103 _step‚459‚ ߊÌˆÆ & lr-AdamW/pg1‚0.0008158887773664137 & lr-AdamW/pg2‚0.0008158887773664137 trainer/global_step‚5699 % _timestamp‚1.7843707579497998e+09 _runtime‚2699.393617103 _step‚459ËÂ`¨¡2‰- _wandbruntime‚2699  _runtime‚2699NpÀÐ'ŠÊO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96919819 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217953325290706 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚460 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54124453 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218268075380736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3364818108928 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218268058341376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96929798 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 % _timestamp‚1.7843707579508638e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42805343 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221632876450304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218268075380736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54124453 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚2699.394865749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54114476 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3476423810048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3352985859024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26292308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12957269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217953222114256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214600236255232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3353071996114 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205412759568384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3364906045952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221632876450304 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42805345 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214600253294592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3364906045952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12957501 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42805345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42805343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208889361779712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218268058341376LM¼Yó‹ÏO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39249809 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3476602211328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221632981426688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217953325290706 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚460 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54124453 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218268075380736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3364818108928 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218268058341376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96929798 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@Dr„ÁâeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 % _timestamp‚1.7843707579508638e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42805343 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221632876450304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218268075380736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54124453 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚2699.394865749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54114476 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3476423810048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3352985859024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26292308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12957269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217953222114256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214600236255232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3353071996114 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205412759568384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3364906045952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221632876450304 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42805345 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214600253294592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3364906045952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12957501 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42805345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42805343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208889361779712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218268058341376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96919819 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96946314 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205447646179328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214636725329920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚461 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚5699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3365387468288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26296792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39256216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205447671083008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221670661757440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217990364647676 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221670661757440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214636708290560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217990261471114 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54123757 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚2699.740699619 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54133748 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3365387468288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12959427 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96936321 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26296789 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218305186344960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218305169305600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3365475412480•%pß&ŽÙM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3365475412480 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54133748 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚461 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚5699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3365387468288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26296792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39256216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205447671083008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221670661757440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217990364647676 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221670661757440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214636708290560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217990261471114 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54123757 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚2699.740699619 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54133748 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3365387468288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12959427 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96936321 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26296789 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218305186344960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218305169305600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3365475412480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12959669 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3477013386240 % _timestamp‚1.7843707582966626e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54123757 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96946314 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208924659565568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96946314 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205447646179328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214636725329920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 P¦ ‡ 0loss/FRt1/assignment_loss‚1.7366760308709734 3loss/FBt2/assignment_loss‚0.0005703407833088146 0loss/FBt2/detection_loss‚0.44039193564094603 trainer/global_step‚5699 3loss/SRqqt2/assignment_loss‚0.45024139540277414 1loss/SRqqt1/detection_loss‚0.9345787231868599 0loss/FRt2/assignment_loss‚0.6631978899240494 2loss/SRqqt1/assignment_loss‚0.6354785269477361 /loss/FRt1/detection_loss‚0.6480334445981023 /loss/FRt2/detection_loss‚1.5073069834098634 1loss/SRqqt2/detection_loss‚0.6444461756591621 _step‚462 3loss/FBt1/assignment_loss‚0.0005805925005095402 /loss/FBt1/detection_loss‚0.4708279621503923 %loss/total_loss‚8.132330001074678 epoch‚7 % _timestamp‚1.7843707582984715e+09 _runtime‚2699.742007458`Ûl‘Œ epoch‚7 % _timestamp‚1.7843707582984715e+09 _runtime‚2699.742007458 0loss/FRt1/assignment_loss‚1.7366760308709734 3loss/FBt2/assignment_loss‚0.0005703407833088146 0loss/FBt2/detection_loss‚0.44039193564094603 trainer/global_step‚5699 3loss/SRqqt2/assignment_loss‚0.45024139540277414 1loss/SRqqt1/detection_loss‚0.9345787231868599 0loss/FRt2/assignment_loss‚0.6631978899240494 2loss/SRqqt1/assignment_loss‚0.6354785269477361 /loss/FRt1/detection_loss‚0.6480334445981023 /loss/FRt2/detection_loss‚1.5073069834098634 1loss/SRqqt2/detection_loss‚0.6444461756591621 _step‚462 3loss/FBt1/assignment_loss‚0.0005805925005095402 /loss/FBt1/detection_loss‚0.4708279621503923 %loss/total_loss‚8.132330001074678Îñi¡œ’‚(:‘ ƬíÒ›¿ò gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿uÎÇ“‚(:¼ ƬíÒ”²É‘% disk.nvme0n1.out‚4811.80126953125' disk.nvme0c0n1.out‚4811.80126953125 network.sent‚ 7.591251e+06) disk./.usagePercent‚39.61482755086727 disk.nvme0n1.in‚3607.25 disk.sda1.in‚0 disk.dm-1.out‚ 65.08203125 disk.dm-0.in‚3607.25# memory_percent‚98.8060712814331 cpu‚26.951439521527014 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0" disk.dm-0.out‚4811.80126953125( proc.memory.availableMB‚ 195.61328125 disk.nvme0c0n1.in‚3607.25$ disk./.usageGB‚590.2895584106445 disk.sda.out‚ 65.08203125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2689.15234375 network.recv‚878551 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 65.08203125 disk.rbd1.in‚0 disk.rbd1.out‚6.5625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375* proc.memory.percent‚16.413283348083496äöüV‘”‚(:† Õ¬íÒûÿŠ gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚182.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òfzÄ•‚(:¹ Õ¬íÒ¶þ‹ disk.dm-1.out‚ 65.23828125 disk.dm-0.in‚3607.25 disk.drbd2165.in‚0* proc.memory.percent‚16.413283348083496 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.sda3.out‚ 65.23828125 disk.sda.in‚0.109375$ memory_percent‚98.80726337432861 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚39.614831134972746$ disk./.usageGB‚590.2896118164062 disk.sda2.out‚0 disk.sda.out‚ 65.23828125 disk.drbd2165.out‚0 disk.rbd1.out‚6.625 network.sent‚ 7.641014e+06 disk.sda1.out‚0! disk.dm-0.out‚4819.3251953125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚3607.25 disk.nvme0n1.in‚3607.25 cpu‚26.94753574631914 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚4819.3251953125# proc.memory.rssMB‚ 2689.15234375& disk.nvme0c0n1.out‚4819.3251953125 proc.cpu.threads‚17( proc.memory.availableMB‚ 195.41796875 network.recv‚884194shº2–- _wandbruntime‚2721  _runtime‚2721òyXÇ—Á trainer/global_step‚5749 % _timestamp‚1.7843707799932783e+09 _runtime‚2721.437067919 _step‚463 & lr-AdamW/pg1‚0.0008152372373335291 & lr-AdamW/pg2‚0.00081523723733352915†M Ì˜Æ & lr-AdamW/pg2‚0.0008152372373335291 trainer/global_step‚5749 % _timestamp‚1.7843707799932783e+09 _runtime‚2721.437067919 _step‚463 & lr-AdamW/pg1‚0.0008152372373335291Ï–¾å2™- _wandbruntime‚2721  _runtime‚2721]™äÅÚÉO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207158335299584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3381383838732 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚210664409124864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚210664203722752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3393406270976 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚464 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocat«¥P "ed‚ 3381469975822 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97746771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39582284 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43166395 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3393406270976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223516924886528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43166393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223517029862912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223516924886528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54580378 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207158310395904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚220123623591936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 $ _timestamp‚1.784370779994327e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216423838021632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220123606552576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13066014 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54590355 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219805221860364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223517029862912 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219805325036814 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13065779 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216423855060992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3393318333952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39582522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26516508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3505893326848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166395˜±¹9Ô'›ÎO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚220123606552576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97756750 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166393 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54580378 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚220123623591936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26516505 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223516924886528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43166393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223517029862912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223516924886528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54580378 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207158310395904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚220123623591936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 $ _timestamp‚1.784370779994327e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216423838021632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220123606552576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13066014 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54590355 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219805221860364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223517029862912 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219805325036814 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13065779 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216423855060992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3393318333952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39582522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26516508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3505893326848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166395 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207158335299584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3381383838732 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚210664409124864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚210664203722752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3393406270976 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚464 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3381469975822 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97746771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39582284 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43166395 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3393406270976 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97773266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97763273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97763273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216460327096320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚220160734556160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54599650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207193221910528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3393975637504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223554710193664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54589659 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26520992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚220160717516800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3381951160262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210699704522752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43173614 _runtime‚2721.780332061 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13068170 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39589162 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223554710193664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207193246814208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3393887693312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216460310056960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3382037297464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97773266 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223554605210112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3393887693312 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39588912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219842261217222 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0GIièñžÙM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚220160734556160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54599650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207193221910528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3393975637504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223554710193664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54589659 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26520992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚220160717516800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3381951160262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210699704522752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43173614 _runtime‚2721.780332061 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13068170 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39589162 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223554710193664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207193246814208 4.DeviceStat/áf îsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3393887693312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216460310056960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3382037297464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97773266 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223554605210112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3393887693312 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39588912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219842261217222 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26520989 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 % _timestamp‚1.7843707803362665e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97773266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97763273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97763273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216460327096320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327Ñ~P×”2Ÿ- _wandbruntime‚2721  _runtime‚27212ùŠà ‚ 0loss/FRt2/assignment_loss‚0.7780575915824536 2loss/FBt2/assignment_loss‚0.006260778234395202 1loss/SRqqt1/detection_loss‚0.7827299275827375 epoch‚7 0loss/FRt1/assignment_loss‚1.8420512644205806 2loss/SRqqt2/assignment_loss‚0.4999384954236901 0loss/FBt2/detection_loss‚0.37403390226453276 $loss/total_loss‚8.28379199644736 2loss/SRqqt1/assignment_loss‚0.6511540364484714 /loss/FRt2/detection_loss‚1.4442964343224154 1loss/SRqqt2/detection_loss‚0.6426992310516009 _runtime‚2721.781685375 _step‚466 1loss/FBt1/assignment_loss‚0.01372635403123962 /loss/FRt1/detection_loss‚0.5981344644329901 /loss/FBt1/detection_loss‚0.6507095166522524 trainer/global_step‚5749 % _timestamp‚1.7843707803381455e+09ïÓm ¡‡ 1loss/FBt1/assignment_loss‚0.01372635403123962 /loss/FRt1/detection_loss‚0.5981344644329901 /loss/FBt1/detection_loss‚0.6507095166522524 trainer/global_step‚5749 % _timestamp‚1.7843707803381455e+09 0loss/FRt2/assignment_loss‚0.7780575915824536 2loss/FBt2/assignment_loss‚0.006260778234395202 1loss/SRqqt1/detection_loss‚0.7827299275827375 epoch‚7 0loss/FRt1/assignment_loss‚1.8420512644205806 2loss/SRqqt2/assignment_loss‚0.4999384954236901 0loss/FBt2/detection_loss‚0.37403390226453276 $loss/total_loss‚8.28379199644736 2loss/SRqqt1/assignment_loss‚0.6511540364484714 /loss/FRt2/detection_loss‚1.4442964343224154 1loss/SRqqt2/detection_loss‚0.6426992310516009 _runtime‚2721.781685375 _step‚466ÒU‹Ì›¢‚(: ä¬íÒù¤¼Š gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚178.858' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.77160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ÓKãÕ£‚(:Ê ä¬íÒ¾þäŒ disk.sda.out‚ 65.57421875 disk.drbd2165.out‚0 disk.rbd1.out‚6.625 disk.sda1.in‚0 cpu‚26.94363443788759 disk.sda3.out‚ 65.57421875 disk.nvme0n1.in‚ 3664.4453125* proc.memory.percent‚16.413283348083496 disk.sda3.in‚0.109375" disk.nvme0c0n1.in‚ 3664.4453125% disk.nvme0n1.out‚4822.05322265625( proc.memory.availableMB‚ 195.55859375 disk.sda2.out‚0" disk.dm-0.out‚4822.05322265625 proc.cpu.threads‚17) disk./.usagePercent‚39.61483267101794 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚98.80640506744385 network.sent‚ 7.645998e+06 network.recv‚887393 disk.dm-1.out‚ 65.57421875 disk.dm-0.in‚ 3664.4453125' disk.nvme0c0n1.out‚4822.05322265625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2689.15234375$ disk./.usageGB‚590.2896347045898 disk.dm-1.in‚0.109375 disk.sda.in‚0.1093752¿‰2¤- _wandbruntime‚2743  _runtime‚2743~wS8Ç¥Á & lr-AdamW/pg1‚0.0008145794313284377 & lr-AdamW/pg2‚0.0008145794313284377 trainer/global_step‚5799 % _timestamp‚1.7843708018243525e+09 _runtime‚2743.268142075 _step‚467 3˜3Ì¦Æ & lr-AdamW/pg1‚0.0008145794313284377 & lr-AdamW/pg2‚0.0008145794313284377 trainer/global_step‚5799 % _timestamp‚1.7843708018243525e+09 _runtime‚2743.268142075 _step‚467ÓFê&Ã2§- _wandbruntime‚2743  _runtime‚2743xÿ·Ð'¨ÊO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚468 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26740708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3535544734208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221979154763776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43527443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55056257 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39914582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55046280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225400973322752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3535370527232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843708018253927e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225400973322752 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218247456827392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55056257 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3421906496000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221979171803136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218247439788032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212439256654336 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 _runtime‚2743.269542939 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43527443 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221979171803136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208903911030784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208903886127104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221657221606472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3421818558976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26740705 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192’â[ÔÕ'©ÏO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221979171803136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208903911030784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208903886127104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221657221606472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3421818558976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26740705 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚468 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26740708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3535544734208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221979154763776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43527443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55056257 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39914582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55046280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225400973322752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3535370527232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843708018253927e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225400973322752 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218247456827392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55056257 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3421906496000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221979171803136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218247439788032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212439256654336 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 _runtime‚2743.269542939 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43527443ÔÄj½2ª- _wandbruntime‚2743  _runtime‚2743@¡N+ «ÔM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3422475862528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13175994 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.oˆ¿`;¯n_train_batch_end/active_bytes.all.allocated‚225438758629888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55055561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221694260963330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26745189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2743.614130926 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚222016282767360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3410435277172 % _timestamp‚1.7843708021700022e+09 _step‚469 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43534666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98600218 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55065552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222016265728000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39921413 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3535959370240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39921183 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222016265728000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218283911823360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26745192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43534666 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98590225 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43534664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3422475862528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98600218 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208938822545408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212474757011968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208938797641728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212474956115456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3422387918336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55065552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230Ýd“ß&¬ÙM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208938822545408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212474757011968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208938797641728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212474956115456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3422387918336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55065552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3422475862528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13175994 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225438758629888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55055561 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221694260963330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26745189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2743.614130926 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚222016282767360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3410435277172 % _timestamp‚1.7843708021700022e+09 _step‚469 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43534666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98600218 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55065552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222016265728000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39921413 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3535959370240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39921183 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222016265728000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218283911823360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26745192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43534666 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98590225 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43534664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3422475862528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98600218Õ?ߧ2­- _wandbruntime‚2743  _runtime‚2743˜<Þ®€ _runtime‚2743.615427664 0loss/FRt1/assignment_loss‚1.7157781306367654 2loss/SRqqt1/assignment_loss‚0.7056213667439446 2loss/SRqqt2/assignment_loss‚0.5052167253143125 /loss/FRt2/detection_loss‚1.6397213248125408 1loss/SRqqt2/detection_loss‚0.5714443779619994 % _timestamp‚1.7843708021718729e+09 $loss/total_loss‚8.87495846480297 epoch‚7 2loss/FBt1/assignment_loss‚0.020447906086420678 /loss/FRt1/detection_loss‚0.5617581391192009 1loss/SRqqt1/detection_loss‚0.9463091468846894 /loss/FBt1/detection_loss‚0.8081867439562783 _step‚470 0loss/FRt2/assignment_loss‚0.8300328139335879 1loss/FBt2/assignment_loss‚0.10574308041365772 .loss/FBt2/detection_loss‚0.464698708939573 trainer/global_step‚5799y·Xæ ¯… 2loss/FBt1/assignment_loss‚0.020447906086420678 /loss/FRt1/detection_loss‚0.5617581391192009 1loss/SRqqt1/detection_loss‚0.9463091468846894 /loss/FBt1/detection_loss‚0.8081867439562783 _step‚470 0loss/FRt2/assignment_loss‚0.8300328139335879 1loss/FBt2/assignment_loss‚0.10574308041365772 .loss/FBt2/detection_loss‚0.464698708939573 trainer/global_step‚5799 _runtime‚2743.615427664 0loss/FRt1/assignment_loss‚1.7157781306367654 2loss/SRqqt1/assignment_loss‚0.7056213667439446 2loss/SRqqt2/assignment_loss‚0.5052167253143125 /loss/FRt2/detection_loss‚1.6397213248125408 1loss/SRqqt2/detection_loss‚0.5714443779619994 % _timestamp‚1.7843708021718729e+09 $loss/total_loss‚8.87495846480297 epoch‚7ÖsЬ‘°‚(:† ó¬íÒûöåŒ gpu.0.gpu‚62.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚173.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÒÞàб‚(:Å ó¬íÒ宦 disk.rbd0.in‚0' disk.nvme0c0n1.out‚4827.21142578125 disk.dm-1.out‚ 65.8515625$ memory_percent‚98.80504608154297% proc.memory.availableMB‚ 195.78125 disk.drbd2165.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚590.2897453308105 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 cpu‚26.9398311404124" disk.nvme0c0n1.in‚ 3664.4453125% disk.nvme0n1.out‚4827.21142578125 disk.sda2.in‚0 disk.drbd2165.out‚0 network.sent‚ 7.708886e+06 network.recv‚892947 disk.sda3.in‚0.109375 disk.rbd0.out‚0 disk.dm-0.in‚ 3664.4453125 disk.nvme0n1.in‚ 3664.4453125 proc.cpu.threads‚17 disk.rbd1.out‚6.6875* disk./.usagePercent‚39.614840095236445 disk.sda3.out‚ 65.8515625 disk.sda.in‚0.109375 disk.sda.out‚ 65.8515625 disk.sda1.out‚0" disk.dm-0.out‚4827.21142578125# proc.memory.rssMB‚ 2689.15234375* proc.memory.percent‚16.413283348083496™~‘²‚(:† ‚­íÒ×äÔŠ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚128.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`~Qг‚(:Å ‚­íÒó²ÁŒ network.recv‚898073 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 66.12109375" memory_percent‚98.807692527771% disk.nvme0c0n1.out‚4834.728515625 disk.rbd0.in‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 66.12109375 disk.sda.in‚0.109375 network.sent‚ 7.771646e+06* disk./.usagePercent‚39.614825246799455# disk.nvme0n1.out‚4834.728515625 disk.dm-0.in‚ 3664.4453125 disk.sda.out‚ 66.12109375* proc.memory.percent‚16.413283348083496 cpu‚26.936265108097125" disk.nvme0c0n1.in‚ 3664.4453125 disk.nvme0n1.in‚ 3664.4453125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.out‚4834.728515625 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚6.6875( proc.memory.availableMB‚ 195.34765625$ disk./.usageGB‚590.2895240783691 disk.sda2.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2689.15234375P²ª2´- _wandbruntime‚2765  _runtime‚2765y¾ÓùǵÁ _step‚471 & lr-AdamW/pg1‚0.0008139153698872377 & lr-AdamW/pg2‚0.0008139153698872377 trainer/global_step‚5849 % _timestamp‚1.7843708237963398e+09 _runtime‚2765.240098828ËS±ŠÌ¶Æ _runtime‚2765.240098828 _step‚471 & lr-AdamW/pg1‚0.0008139153698872377 & lr-AdamW/pg2‚0.0008139153698872377 trainer/global_step‚5849 % _timestamp‚1.7843708237963398e+09×$m+à2·- _wandbruntime‚2765  _runtime‚2765]ÉתÐ'¸ÊO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214214290274304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55522159 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223834720014336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43888493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3450318784000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214214495676416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3450406721024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26964905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13282874 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227285126735360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43888495 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55512182 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 trainer/global_step‚5849 _runtime‚2765.241358036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26964908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220071041554432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40248029 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43888493 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55512182 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223509221352580 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3438265935238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223834702974976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13283121 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3564828416000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227285021758976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40247779 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227285021758976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223509324529030 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220071058593792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2M{§=¹ÏO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55522159 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55522159 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223834720014336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43888493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3450318784000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214214495676416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3450406721024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26964905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13282874 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227285126735360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43888495 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55512182 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 trainer/global_step‚5849 _runtime‚2765.241358036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26964908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220071041554432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40248029 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43888493 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55512182 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223509221352580 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3438265935238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223834702974976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13283121 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3564828416000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227285021758976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40247779 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227285021758976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223509324529030 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220071058593792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214214290274304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _runtime‚2765.585121395 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3450888143360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99427170 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3450888143360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227322807066112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214249995252736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223546260709438 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚5849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223871830978560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13284988 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚210684398276608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3438833256880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43895714 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55531454 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40254377 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26969392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _step‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43895716 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55521463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13285233 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3565416484864 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223546363886000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3438747119678 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99427170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214249789857792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843708241410851e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55521463 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43895716ýº@ß&¼ÙM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223871813939200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _runtime‚2765.585121395 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3450888143360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99427170 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3450888143360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227322807066112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214249995252736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223546260709438 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚5849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223871830978560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13284988 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚210684398276608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3438833256880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43895714 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55531454 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40254377 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26969392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _step‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43895716 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55521463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13285233 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3565416484864 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223546363886000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3438747119678 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99427170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214249789857792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843708241410851e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55521463 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43895716 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55531454Ù]X „2½- _wandbruntime‚2765  _runtime‚2765céé¾ 2loss/SRqqt2/assignment_loss‚0.5645492479850671 4loss/FBt2/assignment_loss‚4.9704376187767586e-05 /loss/FRt1/detection_loss‚0.5508944668624924 1loss/SRqqt1/detection_loss‚0.8767082670987663 epoch‚7 trainer/global_step‚5849 2loss/SRqqt1/assignment_loss‚0.7349273591679862 .loss/FRt2/detection_loss‚1.508395709105852 %loss/total_loss‚8.452293470881063 _runtime‚2765.586601277 1loss/FBt1/assignment_loss‚0.00748008498027275 $ _timestamp‚1.784370824143042e+09 _step‚474 0loss/FRt1/assignment_loss‚1.7667104554773525 1loss/SRqqt2/detection_loss‚0.6003028323175386 /loss/FBt1/detection_loss‚0.5685808138389673 .loss/FBt2/detection_loss‚0.719300887071649 0loss/FRt2/assignment_loss‚0.5543936425989324]B9½ ¿† .loss/FRt2/detection_loss‚1.508395709105852 %loss/total_loss‚8.452293470881063 _runtime‚2765.586601277 1loss/FBt1/assignment_loss‚0.00748008498027275 $ _timestamp‚1.784370824143042e+09 _step‚474 0loss/FRt1/assignment_loss‚1.7667104554773525 1loss/SRqqt2/detection_loss‚0.6003028323175386 /loss/FBt1/detection_loss‚0.5685808138389673 .loss/FBt2/detection_loss‚0.719300887071649 0loss/FRt2/assignment_loss‚0.5543936425989324 2loss/SRqqt2/assignment_loss‚0.5645492479850671 4loss/FBt2/assignment_loss‚4.9704376187767586e-05 /loss/FRt1/detection_loss‚0.5508944668624924 1loss/SRqqt1/detection_loss‚0.8767082670987663 epoch‚7 trainer/global_step‚5849 2loss/SRqqt1/assignment_loss‚0.7349273591679862Ú‰Û”5‘À‚(:† ‘­íÒ¢ý Š gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜŒ™hÁ‚(:Å ‘­íҪɎŽ disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚6.75 disk.sda3.out‚ 66.26171875 disk.sda.out‚ 66.26171875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 3664.4453125 network.sent‚ 7.776588e+06 network.recv‚901702 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 3664.4453125! disk.dm-0.out‚4838.6240234375& disk.nvme0c0n1.out‚4838.6240234375 proc.cpu.threads‚17) disk./.usagePercent‚39.61482806288233 disk.dm-1.out‚ 66.26171875" disk.nvme0c0n1.in‚ 3664.4453125# proc.memory.rssMB‚ 2689.15234375* proc.memory.percent‚16.413283348083496$ disk./.usageGB‚590.2895660400391 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 195.359375 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0.109375 disk.rbd1.in‚0ºÒÄh$ disk.nvme0n1.out‚4838.6240234375$ memory_percent‚98.80762100219727 cpu‚26.932471034742022×ÛÑ+2Â- _wandbruntime‚2787  _runtime‚2787ý…6ÇÃÁ _runtime‚2787.050821799 _step‚475 & lr-AdamW/pg1‚0.0008132450636462211 & lr-AdamW/pg2‚0.0008132450636462211 trainer/global_step‚5899 % _timestamp‚1.7843708456070302e+09OVvÌÄÆ % _timestamp‚1.7843708456070302e+09 _runtime‚2787.050821799 _step‚475 & lr-AdamW/pg1‚0.0008132450636462211 & lr-AdamW/pg2‚0.0008132450636462211 trainer/global_step‚5899Û,»12Å- _wandbruntime‚2787  _runtime‚27875ñJÓ'ÆÍO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13391647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212395062493184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚225361324275138 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215989366856704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2787.052056348 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55978084 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3478906946048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44249543 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27189105 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215989561773056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225690268225536 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100237606 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3594329267200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13391414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229169070195200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229169175171584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55978084 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212395037589504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3466663914946 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225690268225536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3478819009024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225690251186176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5899 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229169070195200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229169175171584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55978084 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212395037589504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3466663914946 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225690268225536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3478819009024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225690251186176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5899 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090Ü®î÷O2È- _wandbruntime‚2787  _runtime‚2787 óQÝ&É×M NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55997356 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186520576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229206855502336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216025059371008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3467231236588 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55987365 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225727379189760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3479388368384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100254122 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3479476312576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225727362150400 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100244129 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27193592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55997356 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225398363632108 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55987365 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13393781 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3595085363200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3594923746304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229206750518784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3479476312576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44256764 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40587373 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44256766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3467145099386 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100254122 trainer/global_step‚5899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100244129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229206855502336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225727379189760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40587153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚221931115356160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225398260455546 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44256766 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212429974007808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 $ _timestamp‚1.784370845948006e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚221931132395520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216024872850432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146:üEŠ¡ÊÜM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216024872850432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55997356 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186520576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229206855502336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216025059371008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3467231236588 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55987365 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225727379189760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3479388368384 L>DeviceStatsMonitor.on_train_óȼA batch_end/allocation.all.allocated‚ 100254122 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3479476312576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225727362150400 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100244129 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27193592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55997356 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225398363632108 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55987365 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13393781 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3595085363200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3594923746304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229206750518784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3479476312576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44256764 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40587373 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44256766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3467145099386 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100254122 trainer/global_step‚5899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100244129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229206855502336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225727379189760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40587153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚221931115356160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225398260455546 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44256766 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212429974007808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 $ _timestamp‚1.784370845948006e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚221931132395520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473ÝÚ™n2Ë- _wandbruntime‚2787  _runtime‚2787Z@3Ì€ 2loss/FBt1/assignment_loss‚0.004531612951251418 /loss/FRt1/detection_loss‚0.6165434877574418 0loss/SRqqt2/detection_loss‚0.689761225103184 trainer/global_step‚5899 $ _timestamp‚1.784370845949654e+09 _runtime‚2787.393189353 0loss/FRt1/assignment_loss‚1.6329063598872626 1loss/SRqqt1/assignment_loss‚0.727869756468127 3loss/FBt2/assignment_loss‚0.0029684292282865757 /loss/FBt1/detection_loss‚1.0273309923708438 epoch‚7 _step‚478 0loss/FRt2/assignment_loss‚0.8378312046350871 /loss/FRt2/detection_loss‚1.5859679637891626 1loss/SRqqt1/detection_loss‚0.8881385860326334 2loss/SRqqt2/assignment_loss‚0.5176885219110074 .loss/FBt2/detection_loss‚0.371016580482117 %loss/total_loss‚8.902554720616404Aòâ Í… 2loss/FBt1/assignment_loss‚0.004531612951251418 /loss/FRt1/detection_loss‚0.6165434877574418 0loss/SRqqt2/detection_loss‚0.689761225103184 trainer/global_step‚5899 $ _timestamp‚1.784370845949654e+09 _runtime‚2787.393189353 0loss/FRt1/assignment_loss‚1.6329063598872626 1loss/SRqqt1/assignment_loss‚0.727869756468127 3loss/FBt2/assignment_loss‚0.0029684292282865757 /loss/FBt1/detection_loss‚1.0273309923708438 epoch‚7 _step‚478 0loss/FRt2/assignment_loss‚0.8378312046350871 /loss/FRt2/detection_loss‚1.5859679637891626 1loss/SRqqt1/detection_loss‚0.8881385860326334 2loss/SRqqt2/assignment_loss‚0.5176885219110074 .loss/FBt2/detection_loss‚0.371016580482117 %loss/total_loss‚8.902554720616404Þš]W՜΂(:‘  ­íÒýÒ±Š gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚165.361' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.072199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ÈÐÏ‚(:Å  ­íÒï·ðŒ# disk./.usageGB‚590.289665222168! disk.dm-0.out‚4844.3037109375 disk.drbd2165.out‚0 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375 disk.sda.out‚ 66.56640625 proc.cpu.threads‚17$ disk.nvme0n1.out‚4844.3037109375 network.recv‚906871 disk.rbd1.out‚6.75" disk.nvme0c0n1.in‚ 3730.8046875 disk.sda1.out‚0 disk.dm-0.in‚ 3730.8046875 disk.sda3.out‚ 66.56640625 network.sent‚ 7.83963e+06 disk.drbd2165.in‚0$ memory_percent‚98.80719184875488* proc.memory.percent‚16.413283348083496 disk.sda2.in‚0& disk.nvme0c0n1.out‚4844.3037109375 disk.nvme0n1.in‚ 3730.8046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 66.56640625 disk.sda3.in‚0.109375# proc.memory.rssMB‚ 2689.15234375* disk./.usagePercent‚39.614834719078225 disk.sda1.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 195.4296875 cpu‚26.928773715738732|‘Ђ(:† ¯­íÒÝÂŒ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑHÓÑ‚(:È ¯­íÒ¶¼­ disk.rbd1.out‚6.8125% disk.nvme0n1.out‚4851.72216796875 disk.sda2.out‚0" disk.dm-0.out‚4851.72216796875# proc.memory.rssMB‚ 2689.15234375 network.recv‚911947 disk.rbd1.in‚0' disk.nvme0c0n1.out‚4851.72216796875 disk.rbd0.in‚0 disk.dm-0.in‚ 3730.8046875* proc.memory.percent‚16.413283348083496 disk.dm-1.out‚ 66.8203125$ disk./.usageGB‚590.2893218994141" disk.nvme0c0n1.in‚ 3730.8046875 disk.sda2.in‚0 disk.sda3.out‚ 66.8203125' proc.memory.availableMB‚ 195.4453125 network.sent‚ 7.902353e+06 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚39.614811678400145 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚0.109375$ memory_percent‚98.80709648132324 proc.cpu.threads‚17 disk.dm-1.in‚0.109375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3730.8046875 disk.sda3.in‚0.109375 disk.sda.out‚ 66.8203125 cpu‚26.92508146834523¶£e2Ò- _wandbruntime‚2808  _runtime‚2808Tèß|ÆÓÀ _runtime‚2808.934036132 _step‚479 & lr-AdamW/pg1‚0.0008125685233417028 & lr-AdamW/pg2‚0.0008125685233417028 trainer/global_step‚5949 $ _timestamp‚1.784370867490239e+09³qDbËÔÅ _runtime‚2808.934036132 _step‚479 & lr-AdamW/pg1‚0.0008125685233417028 & lr-AdamW/pg2‚0.0008125685233417028 trainer/global_step‚5949 $ _timestamp‚1.784370867490239e+09ßâwÉ2Õ- _wandbruntime‚2808  _runtime‚2808DÁÚÔ'ÖÎO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44610593 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27413305 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227213324021246 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27413308 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3624047566848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227545816436736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227545799397376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227545799397376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 _step‚480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843708674913237e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40913249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13499941 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231053118631424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3507319234048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44610595 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217764480389120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56443986 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3494975757564 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56453963 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214140613320704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214140638224384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231053223607808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3507319234048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚227213220844796 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101054579 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56453963 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227545816436736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56443986 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3623867068416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44610595 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3507407171072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚2808.935308193 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231053223607808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101064558 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44610593 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ƒÃž…$×ÓO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231053118631424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40913249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13499941 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231053118631424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3507319234048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44610595 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217764480389120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56443986 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3494975757564 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56453963 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214140613320704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214140638224384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231053223607808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3507319234048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚227213220844796 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101054579 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56453963 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227545816436736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56443986 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3623867068416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44610595 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3507407171072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚2808.935308193 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231053223607808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101064558 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44610593 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44610593 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27413305 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227213324021246 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27413308 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3624047566848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227545816436736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227545799397376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227545799397376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 _step‚480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843708674913237e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UGDeviceStatsMonitor.on_train_batch_start/alloca•7¥8Xted_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56453267 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚481 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3507976537600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101071081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101081074 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231090798955008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56453267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27417789 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227250260201654 _runtime‚2809.274678125 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44617814 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843708678306155e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227582927400960 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3507888593408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3624634530816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44617816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40919608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223754717122560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40919851 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101071081 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3507976537600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227582927400960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3624451942400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223754734161920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101081074 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 trainer/global_step‚5949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56463258 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227582910361600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3495629216296 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231090903938560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13502059 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214175549739008Ü(â¸ã&ÚÝM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231090903938560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13502059 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214175549739008 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227250363378216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56463258 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231090903938560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217800184269824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214175524835328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3495543079094 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3507888593408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217799976777728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56453267 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚481 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3507976537600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101071081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101081074 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231090798955008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56453267 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27417789 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227250260201654 _runtime‚2809.274678125 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44617814 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843708678306155e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227582927400960 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3507888593408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3624634530816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44617816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40919608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223754717122560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40919851 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101071081 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3507976537600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227582927400960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3624451942400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223754734161920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101081074 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 trainer/global_step‚5949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56463258 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227582910361600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3495629216296áÄÿ|Y2Û- _wandbruntime‚2809  _runtime‚2809n‚JµÜ /loss/FRt1/assignment_loss‚1.746754968645929 3loss/FBt1/assignment_loss‚5.528820646985434e-05 1loss/FBt2/assignment_loss‚0.13577697363324895 epoch‚7 $ _timestamp‚1.784370867832184e+09 _runtime‚2809.275718238 0loss/FRt2/assignment_loss‚0.7506719303772589 2loss/SRqqt1/assignment_loss‚0.5982939740355886 1loss/SRqqt1/detection_loss‚0.8271100456598973 1loss/SRqqt2/detection_loss‚0.6274470694086969 /loss/FBt1/detection_loss‚0.5665744404707637 trainer/global_step‚5949 2loss/SRqqt2/assignment_loss‚0.5172052610681307 /loss/FRt1/detection_loss‚0.5326461151989199 /loss/FBt2/detection_loss‚0.5588175123710124 %loss/total_loss‚8.440909165248442 /loss/FRt2/detection_loss‚1.5795555861725268 _step‚482€øÝ Ý† /loss/FRt1/assignment_loss‚1.746754968645929 3loss/FBt1/assignment_loss‚5.528820646985434e-05 1loss/FBt2/assignment_loss‚0.13577697363324895 epoch‚7 $ _timestamp‚1.784370867832184e+09 _runtime‚2809.275718238 0loss/FRt2/assignment_loss‚0.7506719303772589 2loss/SRqqt1/assignment_loss‚0.5982939740355886 1loss/SRqqt1/detection_loss‚0.8271100456598973 1loss/SRqqt2/detection_loss‚0.6274470694086969 /loss/FBt1/detection_loss‚0.5665744404707637 trainer/global_step‚5949 2loss/SRqqt2/assignment_loss‚0.5172052610681307 /loss/FRt1/detection_loss‚0.5326461151989199 /loss/FBt2/detection_loss‚0.5588175123710124 %loss/total_loss‚8.440909165248442 /loss/FRt2/detection_loss‚1.5795555861725268 _step‚482⯖î‘Þ‚(:† ¾­íÒ½õ§‹ gpu.0.gpu‚63.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶¹B–Ôß‚(:É ¾­íÒÞàÃŒ network.sent‚ 7.907239e+06 network.recv‚915237 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 67.15234375# memory_percent‚98.8072395324707& proc.memory.availableMB‚ 195.421875 disk.sda1.in‚0* proc.memory.percent‚16.413283348083496 disk.rbd1.out‚6.8125 disk.sda1.out‚0 disk.sda3.out‚ 67.15234375% disk.nvme0n1.out‚4858.02587890625 disk.dm-0.in‚ 3730.8046875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3730.8046875' disk.nvme0c0n1.out‚4858.02587890625 disk.rbd0.out‚0" disk.dm-0.out‚4858.02587890625 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.sda3.in‚0.109375* disk./.usagePercent‚39.614810654370004# disk./.usageGB‚590.289306640625 disk.sda.out‚ 67.15234375 disk.nvme0n1.in‚ 3730.8046875 disk.sda2.out‚0# proc.memory.rssMB‚ 2689.15234375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd0.in‚0 cpu‚26.921629718760425à2à- _wandbruntime‚2830  _runtime‚2830>*ÇáÁ % _timestamp‚1.7843708893285227e+09 _runtime‚2830.772290305 _step‚483 & lr-AdamW/pg1‚0.0008118857598098492 & lr-AdamW/pg2‚0.0008118857598098492 trainer/global_step‚5999J/6éÌâÆ trainer/global_step‚5999 % _timestamp‚1.7843708893285227e+09 _runtime‚2830.772290305 _step‚483 & lr-AdamW/pg1‚0.0008118857598098492 & lr-AdamW/pg2‚0.0008118857598098492ãlty[2ã- _wandbruntime‚2830  _runtime‚2830œõ‰v² äÍO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232937272044032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225541863892992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 $ _timestamp‚1.784370889329606e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229401364647936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229065220590904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56909888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13608695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229065323767354 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56919865 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219539747613696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3653533658112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3523459874362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56909888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3535819459072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229401364647936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44971643 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232937167067648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3535907396096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229401347608576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44971645 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13608931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229401347608576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3653359451136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3523373737272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚484 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3535819459072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101891510 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215886213955584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.a±à»!ll.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3535907396096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41246200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225541846853632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101881531 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44971645 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27637508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56919865 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232937167067648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2830.773624984#ÊFØ'åÒO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101891510 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215886213955584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3535907396096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41246200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225541846853632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101881531 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44971645 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27637508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56919865 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232937167067648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2830.773624984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232937272044032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225541863892992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 $ _timestamp‚1.784370889329606e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229401364647936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229065220590904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56909888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13608695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229065323767354 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56919865 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219539747613696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3653533658112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3523459874362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56909888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3535819459072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229401364647936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44971643 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232937167067648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3535907396096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229401347608576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44971645 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13608931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229401347608576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3653359451136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3523373737272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚484 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3535819459072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327ä æ²–2æ- _wandbruntime‚2831  _runtime‚2831\í Þ&çØM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101898033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232974952374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44978866 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229102363124324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚5999 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41252877 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101898033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚225578335928320 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27641989 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232974952374784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219575249090560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101908026 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3654123620352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13610888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44978866 _runtime‚2831.119311758 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225578318888960 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229102259947762 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229438475612160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101908026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229438458572800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56929160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3536476762624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3536476762624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3524027196004 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44978864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27641992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232974847391232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _step‚485 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13611132 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229438475612160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473Š-¾xã&èÝM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229438458572800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56929160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3536476762624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3536476762624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3524027196004 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44978864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27641992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232974847391232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _step‚485 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13611132 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229438475612160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101898033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232974952374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44978866 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229102363124324 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚5999 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41252877 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101898033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚225578335928320 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27641989 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232974952374784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219575249090560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101908026 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3654123620352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13610888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44978866 _runtime‚2831.119311758 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225578318888960 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229102259947762 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229438475612160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101908026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154åÅJ?-2é- _wandbruntime‚2831  _runtime‚2831jì@DZêƒ 2loss/SRqqt2/assignment_loss‚0.5023482564579617 2loss/FBt1/assignment_loss‚0.027109303192071617 /loss/FBt1/detection_loss‚0.5217767175944412 2loss/SRqqt1/assignment_loss‚0.8408391736601567 2loss/FBt2/assignment_loss‚0.027037843270539343 epoch‚7 trainer/global_step‚5999 /loss/FRt1/detection_loss‚0.6324440940704563 1loss/SRqqt1/detection_loss‚0.8214840782489979 /loss/FBt2/detection_loss‚0.4561717040487565 % _timestamp‚1.7843708896769564e+09 _step‚486 0loss/FRt2/assignment_loss‚0.6903450000393498 /loss/FRt2/detection_loss‚1.5626417274192512 1loss/SRqqt2/detection_loss‚0.6045643626776404 %loss/total_loss‚8.360234ülX214543633 _runtime‚2831.120496253 0loss/FRt1/assignment_loss‚1.6734719538640102)xëˆ 0loss/FRt1/assignment_loss‚1.6734719538640102 2loss/SRqqt2/assignment_loss‚0.5023482564579617 2loss/FBt1/assignment_loss‚0.027109303192071617 /loss/FBt1/detection_loss‚0.5217767175944412 2loss/SRqqt1/assignment_loss‚0.8408391736601567 2loss/FBt2/assignment_loss‚0.027037843270539343 epoch‚7 trainer/global_step‚5999 /loss/FRt1/detection_loss‚0.6324440940704563 1loss/SRqqt1/detection_loss‚0.8214840782489979 /loss/FBt2/detection_loss‚0.4561717040487565 % _timestamp‚1.7843708896769564e+09 _step‚486 0loss/FRt2/assignment_loss‚0.6903450000393498 /loss/FRt2/detection_loss‚1.5626417274192512 1loss/SRqqt2/detection_loss‚0.6045643626776404 %loss/total_loss‚8.360234214543633 _runtime‚2831.120496253æ¤N‰u›ì‚(: Í­íÒêñÈŠ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄØüÈí‚(:½ Í­íÒ¸°Ž" disk.nvme0c0n1.in‚ 3730.8046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 67.390625 disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.rbd1.out‚6.9375# proc.memory.rssMB‚ 2689.15234375 cpu‚26.91807958088091' disk.nvme0c0n1.out‚4864.44482421875 disk.sda1.out‚0! proc.memory.availableMB‚195.5* proc.memory.percent‚16.413283348083496) disk./.usagePercent‚39.61481398246795 disk.dm-1.out‚ 67.390625 disk.sda3.out‚ 67.390625$ disk./.usageGB‚590.2893562316895 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 3730.8046875 disk.nvme0n1.in‚ 3730.8046875 disk.rbd0.in‚0% disk.nvme0n1.out‚4864.44482421875# memory_percent‚98.8067626953125 network.sent‚ 7.982901e+06 network.recv‚920541 disk.sda2.out‚0 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚4864.44482421875§…xÈœî‚(:‘ Ü­íÒåêÌŠ gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚175.529' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.105799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H¥´ÚÈï‚(:½ Ü­íÒ›ÜõŽ disk.sda.in‚0.109375 disk.drbd2165.out‚0* proc.memory.percent‚16.413283348083496 disk.rbd1.out‚6.9375# proc.memory.rssMB‚ 2689.15234375 cpu‚26.914581458445387 network.recv‚925503 disk.rbd0.in‚0 disk.dm-0.out‚ 4867.4765625" proc.memory.availableMB‚130.75$ disk./.usageGB‚590.2890739440918# disk.nvme0c0n1.out‚ 4867.4765625! disk.nvme0n1.in‚ 3765.23046875 disk.rbd0.out‚0) disk./.usagePercent‚39.61479503791042# disk.nvme0c0n1.in‚ 3765.23046875! disk.nvme0n1.out‚ 4867.4765625 disk.dm-0.in‚ 3765.23046875 disk.sda.out‚ 67.70703125 disk.drbd2165.in‚0$ memory_percent‚99.20196533203125 disk.dm-1.out‚ 67.70703125 network.sent‚ 8.032786e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.sda3.out‚ 67.70703125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0.109375…jø 2ð- _wandbruntime‚2852  _runtime‚2852PDÞÀÇñÁ _runtime‚2852.819619126 _step‚487 & lr-AdamW/pg1‚0.0008111967839865038 & lr-AdamW/pg2‚0.0008111967839865038 trainer/global_step‚6049 % _timestamp‚1.7843709113757505e+09-׃ÌòÆ & lr-AdamW/pg1‚0.0008111967839865038 & lr-AdamW/pg2‚0.0008111967839865038 trainer/global_step‚6049 % _timestamp‚1.7843709113757505e+09 _runtime‚2852.819619126 _step‚487çñaG2ó- _wandbruntime‚2852  _runtime‚2852€A‹GÑ'ôËO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27861708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚2852.820905894 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57375790 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227365465659392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234821215503872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57375790 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227365448620032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231256912859136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41578352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41578109 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231256895819776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13716644 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230917220337012 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57385767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13716404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217631789686784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3551771716980 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3551857854070 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45332695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45332695 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234821320480256 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102708483 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3564319684096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564319684096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6049 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221314622916608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217631764783104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231256895819776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45332693 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234821215503872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27861705 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234821320480256 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564407621120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57385767 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231256912859136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221314826221568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102718462 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45332693 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3564407621120k˜©Ö'õÐO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234821320480256 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564407621120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57385767 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231256912859136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221314826221568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102718462 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45332693 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3564407621120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27861708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚2852.820905894 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57375790 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227365465659392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234821215503872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57375790 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227365448620032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231256912859136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41578352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41578109 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231256895819776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13716644 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230917220337012 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57385767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13716404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217631789686784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3551771716980 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3551857854070 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45332695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45332695 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234821320480256 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102708483 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3564319684096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564319684096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6049 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221314622916608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217631764783104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231256895819776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45332693 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234821215503872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27861705轪Š2ö- _wandbruntime‚2853  _runtime‚2853ægiO¡÷ØM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3564976987648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227401937694720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102724985 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6049 _step‚489 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41584984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13718563 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217666676297728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3683444124672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230954259693870 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3564889043456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27866189 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227401920655360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45339914 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3683622518784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45339914 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231294023823360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234858895827456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57385071 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27866192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230954362870432 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41584752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102724985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221350120422400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231294006784000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3552339038510 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UHDeviceStatsMonitor.on_train_batch_end/allocateR„ÛU=d_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45339916 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45339916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3564976987648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231294023823360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2853.161058917 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843709117169366e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102734978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102734978 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚234859000811008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217666701201408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202Q'¹ã&øÝM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102734978 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚234859000811008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217666701201408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3564976987648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227401937694720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102724985 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6049 _step‚489 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41584984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13718563 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217666676297728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3683444124672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230954259693870 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3564889043456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27866189 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227401920655360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45339914 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3683622518784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45339914 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231294023823360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234858895827456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57385071 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27866192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230954362870432 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41584752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102724985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221350120422400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231294006784000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3552339038510 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45339916 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45339916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3564976987648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231294023823360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2853.161058917 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843709117169366e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102734978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327éX#'12ù- _wandbruntime‚2853  _runtime‚2853û…¿ú‚ epoch‚7 2loss/SRqqt1/assignment_loss‚0.6925187828834033 1loss/SRqqt2/assignment_loss‚0.528419216632243 2loss/FBt2/assignment_loss‚0.005662044465358171 1loss/SRqqt2/detection_loss‚0.6724318201975533 .loss/FBt1/detection_loss‚0.575745687664797 0loss/FRt1/assignment_loss‚1.8491226176045976 /loss/FRt1/detection_loss‚0.5908347207727789 /loss/FRt2/detection_loss‚1.6014750020320314 1loss/SRqqt1/detection_loss‚1.0945658386355506 _runtime‚2853.162148453 _step‚490 3loss/FBt1/assignment_loss‚0.0009307053549756361 /loss/FBt2/detection_loss‚0.6620316840037388 %loss/total_loss‚8.993888114438585 trainer/global_step‚6049 % _timestamp‚1.7843709117185998e+09 0loss/FRt2/assignment_loss‚0.72014999419155699!!Ï û‡ 0loss/FRt1/assignment_loss‚1.8491226176045976 /loss/FRt1/detection_loss‚0.5908347207727789 /loss/FRt2/detection_loss‚1.6014750020320314 1loss/SRqqt1/detection_loss‚1.0945658386355506 _runtime‚2853.162148453 _step‚490 3loss/FBt1/assignment_loss‚0.0009307053549756361 /loss/FBt2/detection_loss‚0.6620316840037388 %loss/total_loss‚8.993888114438585 trainer/global_step‚6049 % _timestamp‚1.7843709117185998e+09 0loss/FRt2/assignment_loss‚0.7201499941915569 epoch‚7 2loss/SRqqt1/assignment_loss‚0.6925187828834033 1loss/SRqqt2/assignment_loss‚0.528419216632243 2loss/FBt2/assignment_loss‚0.005662044465358171 1loss/SRqqt2/detection_loss‚0.6724318201975533 .loss/FBt1/detection_loss‚0.575745687664797ê‡Y4–‘ü‚(:† ë­íÒ’Ê°Š gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚189.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S!nÙý‚(:Î ë­íÒ¹¬§ disk.drbd2165.out‚0$ memory_percent‚99.20151233673096( proc.memory.availableMB‚ 130.82421875# proc.memory.rssMB‚ 2689.15234375# disk.nvme0c0n1.in‚ 3765.23046875 disk.rbd0.in‚0 disk.dm-1.in‚0.109375" disk.dm-0.out‚4872.98095703125 network.sent‚ 8.05082e+06' disk.nvme0c0n1.out‚4872.98095703125 disk.sda1.in‚0 disk.sda3.out‚ 68.13671875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 68.13671875 cpu‚26.911079883874283% disk.nvme0n1.out‚4872.98095703125 disk.sda3.in‚0.109375 network.recv‚928994$ disk./.usageGB‚590.2884941101074! disk.nvme0n1.in‚ 3765.23046875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3765.23046875* proc.memory.percent‚16.413283348083496) disk./.usagePercent‚39.61475612476521 disk.drbd2165.in‚0 disk.rbd1.out‚7.0625 proc.cpu.threads‚17 disk.dm-1.out‚ 68.13671875 disk.sda.in‚0.109375l²Ÿ‚2þ- _wandbruntime‚2874  _runtime‚2874¤trBÇÿÁ & lr-AdamW/pg1‚0.0008105016069070131 & lr-AdamW/pg2‚0.0008105016069070131 trainer/global_step‚6099 % _timestamp‚1.7843709334215555e+09 _runtime‚2874.865397952 _step‚491(QNÌ€Æ _step‚491 & lr-AdamW/pg1‚0.0008105016069070131 & lr-AdamW/pg2‚0.0008105016069070131 trainer/global_step‚6099 % _timestamp‚1.7843709334215555e+09 _runtime‚2874.865397952ë¤ë~y2- _wandbruntime‚2874  _runtime‚28741DUSÔ'‚ÎO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3592907846144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41910762 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233112461070336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3592819909120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚492 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223089680359936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57851669 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236705263940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚219377340514304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223089881567744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3712516149760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2874.866778237 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28085908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13824624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233112444030976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843709334227655e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236705263940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236705368916480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 trainer/global_step‚6099 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57851669 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45693745 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57841692 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3592907846144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3580169696688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103545414 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3580255833778 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146ª ëZ ƒÓO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚219377365417984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843709334227655e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236705263940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236705368916480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 trainer/global_step‚6099 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57851669 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45693745 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57841692 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3592907846144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3580169696688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103545414 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3580255833778 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3592907846144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41910762 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233112461070336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3592819909120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚492 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223089680359936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57851669 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236705263940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚219377340514304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223089881567744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3712516149760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.alloca¦pƒðÐ ted‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2874.866778237 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28085908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13824624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233112444030976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57860964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3580737018218 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45700966 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165811200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223125186791936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45700964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41917142 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229225522421760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45700964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13826753 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236743049247232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103551937 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _runtime‚2875.208047185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41917363 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103551937 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103561930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219412276932608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚6099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3593477212672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236743049247232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13826971 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3593389268480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57850973 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233149572034560 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103561930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3593389268480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28090389 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223125377506816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233149554995200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28090392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219412252028928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3712934763008Wÿ˜Ôã&†ÝM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3712934763008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229225539461120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57860964 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3713100574208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3580823155420 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57860964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3580737018218 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45700966 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165811200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223125186791936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45700964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41917142 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229225522421760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45700964 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13826753 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236743049247232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103551937 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _runtime‚2875.208047185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41917363 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103551937 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103561930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219412276932608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚6099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3593477212672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236743049247232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13826971 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3593389268480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57850973 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233149572034560 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103561930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3593389268480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28090389 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223125377506816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233149554995200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28090392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219412252028928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚493íœ,þ2‡- _wandbruntime‚2875  _runtime‚2875Ýžëgˆý .loss/FRt2/detection_loss‚1.429321962385669 _step‚494 2loss/SRqqt1/assignment_loss‚0.6202872019564426 .loss/FRt1/detection_loss‚0.591922064991484 0loss/SRqqt2/detection_loss‚0.641979548020754 /loss/FBt2/detection_loss‚0.6034499221667647 epoch‚7 trainer/global_step‚6099 0loss/FRt1/assignment_loss‚1.7893152861863144 0loss/FRt2/assignment_loss‚0.6796822828231471 2loss/SRqqt2/assignment_loss‚0.4745167630328865 1loss/SRqqt1/detection_loss‚0.8257143584211694 #loss/total_loss‚8.4167822965758 $ _timestamp‚1.784370933765526e+09 2loss/FBt1/assignment_loss‚0.008812220246734151 /loss/FBt1/detection_loss‚0.7438921301315228 _runtime‚2875.209058413 2loss/FBt2/assignment_loss‚0.007888556212912002Zj‰‚ $ _timestamp‚1.784370933765526e+09 2loss/FBt1/assignment_loss‚0.008812220246734151 /loss/FBt1/detection_loss‚0.7438921301315228 _runtime‚2875.209058413 2loss/FBt2/assignment_loss‚0.007888556212912002 .loss/FRt2/detection_loss‚1.429321962385669 _step‚494 2loss/SRqqt1/assignment_loss‚0.6202872019564426 .loss/FRt1/detection_loss‚0.591922064991484 0loss/SRqqt2/detection_loss‚0.641979548020754 /loss/FBt2/detection_loss‚0.6034499221667647 epoch‚7 trainer/global_step‚6099 0loss/FRt1/assignment_loss‚1.7893152861863144 0loss/FRt2/assignment_loss‚0.6796822828231471 2loss/SRqqt2/assignment_loss‚0.4745167630328865 1loss/SRqqt1/detection_loss‚0.8257143584211694 #loss/total_loss‚8.4167822965758î7f 曊‚(: ú­íÒÕݸ‹ gpu.0.gpu‚59.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚129.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"•ë΋‚(:à ú­íÒ¡èÓŒ disk.sda.out‚ 68.35546875 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 130.3359375 network.sent‚ 8.100599e+06 disk.dm-0.out‚ 4876.00390625 disk.sda3.in‚0.109375# disk.nvme0c0n1.in‚ 3765.23046875 disk.rbd0.in‚0 disk.dm-1.in‚0.109375( disk./.usagePercent‚39.6147530526748$ disk./.usageGB‚590.2884483337402$ disk.nvme0c0n1.out‚ 4876.00390625 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚934108 disk.sda3.out‚ 68.35546875 disk.drbd2165.out‚0 disk.rbd1.out‚7.0625$ memory_percent‚99.20449256896973* proc.memory.percent‚16.413283348083496 proc.cpu.threads‚17 disk.dm-0.in‚ 3765.23046875# proc.memory.rssMB‚ 2689.15234375 disk.dm-1.out‚ 68.35546875! disk.nvme0n1.in‚ 3765.23046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0.109375" disk.nvme0n1.out‚ 4876.00390625 cpu‚26.90776814499527ÉôŸ­œŒ‚(:‘ ‰®íÒ–ËÞŠ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÏÕ‚(:Ê ‰®íÒâ³ñ' disk.nvme0c0n1.out‚4882.89794921875% disk.nvme0n1.out‚4882.89794921875 disk.sda1.out‚0 disk.dm-1.in‚0.109375# proc.memory.rssMB‚ 2689.15234375 proc.cpu.threads‚17 disk.sda.out‚ 68.57421875 network.sent‚ 8.163261e+06 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 3765.23046875 disk.sda2.out‚0" disk.dm-0.out‚4882.89794921875 disk.dm-1.out‚ 68.57421875 disk.sda3.out‚ 68.57421875 cpu‚26.904368961660705 disk.sda2.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 3765.23046875 disk.sda.in‚0.109375$ memory_percent‚99.20320510864258 disk.rbd0.out‚0" disk./.usageGB‚590.28857421875 disk.dm-0.in‚ 3765.23046875* proc.memory.percent‚16.413283348083496 network.recv‚939358 disk.rbd1.in‚0 disk.rbd1.out‚7.125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0.109375& proc.memory.availableMB‚ 130.546875) disk./.usagePercent‚39.61476150092343ßq«2Ž- _wandbruntime‚2896  _runtime‚2896Hã'ÇÁ & lr-AdamW/pg1‚0.0008098002397060489 & lr-AdamW/pg2‚0.0008098002397060489 trainer/global_step‚6149 % _timestamp‚1.7843709552156858e+09 _runtime‚2896.659535377 _step‚495^ÍçÖÌÆ & lr-AdamW/pg2‚0.0008098002397060489 trainer/global_step‚6149 % _timestamp‚1.7843709552156858e+09 _runtime‚2896.659535377 _step‚495 & lr-AdamW/pg1‚0.0008098002397060489ïGMˆ2‘- _wandbruntime‚2896  _runtime‚2896´ó1m¿’ÍO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3621408071168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784370955216758e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238589417352704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28310105 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234967992242176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28310108 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58307594 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231012652152832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104372366 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚222 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3741971278336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3741803362816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3621320134144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224864912427520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46054795 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238589312376320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58307594 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13932916 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3608653813486 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238589417352704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2*…œ’Ø'“ÒO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58317571 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3608567676396 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42243021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3621408071168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13933135 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46054793 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238589312376320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221122916245504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚6149 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234967992242176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28310108 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58307594 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231012652152832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104372366 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚222 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3741971278336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3741803362816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3621320134144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224864912427520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46054795 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238589312376320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58307594 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13932916 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3608653813486 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238589417352704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3621408071168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784370955216758e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238589417352704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28310105 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _runtime‚2896.999775853 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3621889493504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104378889 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58326866 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221157852663808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13935218 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3742563497472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235005120245760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6149 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238626992699904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3609221135128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3609134997926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104388882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46062014 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221157827760128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46062014 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58316875 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42249578 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224900219154944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231049124188160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3742391394816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231049141227520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46062016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192j¹6…â–ÜM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58326866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 _step‚497 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _runtime‚2896.999775853 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3621889493504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104378889 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58326866 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221157852663808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13935218 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3742563497472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235005120245760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6149 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238626992699904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3609221135128 XFDeviceStatsMoni µ]Û tor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3609134997926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104388882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46062014 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221157827760128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46062014 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58316875 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42249578 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224900219154944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231049124188160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3742391394816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231049141227520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46062016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46062016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13934989 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ñä2—- _wandbruntime‚2897  _runtime‚2897¢ÿ1 ˜„ trainer/global_step‚6149 _runtime‚2897.001238633 1loss/SRqqt1/detection_loss‚0.9095984676333014 %loss/total_loss‚8.561598926021837 $ _timestamp‚1.784370955557703e+09 _step‚498 2loss/SRqqt2/assignment_loss‚0.4799169027001438 /loss/FRt1/detection_loss‚0.5341432386886901 /loss/FRt2/detection_loss‚1.4473476134663956 1loss/SRqqt2/detection_loss‚0.7134926082760523 /loss/FBt1/detection_loss‚0.6393953247673132 .loss/FBt2/detection_loss‚0.517985951481387 epoch‚7 0loss/FRt1/assignment_loss‚1.8554325377711882 0loss/FRt2/assignment_loss‚0.6910988544875926 4loss/FBt1/assignment_loss‚4.4101615633900856e-05 3loss/FBt2/assignment_loss‚0.0002778903673961016 2loss/SRqqt1/assignment_loss‚0.7728654347667416´\û™‰ 2loss/SRqqt1/assignment_loss‚0.7728654347667416 trainer/global_step‚6149 _runtime‚2897.001238633 1loss/SRqqt1/detection_loss‚0.9095984676333014 %loss/total_loss‚8.561598926021837 $ _timestamp‚1.784370955557703e+09 _step‚498 2loss/SRqqt2/assignment_loss‚0.4799169027001438 /loss/FRt1/detection_loss‚0.5341432386886901 /loss/FRt2/detection_loss‚1.4473476134663956 1loss/SRqqt2/detection_loss‚0.7134926082760523 /loss/FBt1/detection_loss‚0.6393953247673132 .loss/FBt2/detection_loss‚0.517985951481387 epoch‚7 0loss/FRt1/assignment_loss‚1.8554325377711882 0loss/FRt2/assignment_loss‚0.6910988544875926 4loss/FBt1/assignment_loss‚4.4101615633900856e-05 3loss/FBt2/assignment_loss‚0.0002778903673961016ò±œš‚(:‘ ˜®íÒÖôÖ‹ gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÏÿÑÑ›‚(:Æ ˜®íÒÕÔÛŒ disk.dm-0.in‚ 3799.1640625 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚942641& disk.nvme0c0n1.out‚4887.2939453125$ memory_percent‚99.20480251312256# proc.memory.rssMB‚ 2689.15234375 disk.nvme0n1.in‚ 3799.1640625* disk./.usagePercent‚39.614652953728914 disk.dm-1.out‚ 69.0390625! disk.dm-0.out‚4887.2939453125 disk.rbd1.in‚0 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 8.168211e+06$ disk./.usageGB‚590.2869567871094 disk.sda2.in‚0$ disk.nvme0n1.out‚4887.2939453125* proc.memory.percent‚16.413283348083496 cpu‚26.900964721336678 disk.sda2.out‚0 disk.sda3.out‚ 69.0390625 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 130.28515625 disk.sda.out‚ 69.0390625 disk.sda.in‚0.109375 disk.rbd1.out‚7.125" disk.nvme0c0n1.in‚ 3799.1640625,€ó2œ- _wandbruntime‚2918  _runtime‚2918qÓÇÁ _runtime‚2918.702230909 _step‚499 & lr-AdamW/pg1‚0.0008090926936174304 & lr-AdamW/pg2‚0.0008090926936174304 trainer/global_step‚6199 % _timestamp‚1.7843709772584288e+09V§¸©ÌžÆ & lr-AdamW/pg1‚0.0008090926936174304 & lr-AdamW/pg2‚0.0008090926936174304 trainer/global_step‚6199 % _timestamp‚1.7843709772584288e+09 _runtime‚2918.702230909 _step‚499óXi¹2Ÿ- _wandbruntime‚2918  _runtime‚2918ˆÛÕ™Ô' ÎO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28534305 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14040789 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232836253919232 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105199318 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236473322751786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3771446168064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843709772594588e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3636965656104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236823557492736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46415843 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14041027 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3649908296192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2918.707452084 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236823540453376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46415845 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105189339 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240473465788928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58773496 _step‚500 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58783473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236823557492736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222868516880384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3771274058240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46415843 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58783473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240473465788928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236473219575336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240473360812544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3649908296192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42575094 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28534308 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452oé[Ù'¡ÓO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28534305 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14040789 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232836253919232 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105199318 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236473322751786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3771446168064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843709772594588e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3636965656104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236823557492736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46415843 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14041027 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3649908296192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2918.707452084 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236823540453376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46415845 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105189339 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240473465788928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58773496 _step‚500 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58783473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236823557492736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222868516880384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3771274058240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46415843 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58783473 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240473465788928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236473219575336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240473360812544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3649908296192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42575094 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28534308ôÌô„§2¢- _wandbruntime‚2919  _runtime‚2919”GP£ØM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58792768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3772034835968 % _timestamp‚1.7843709776037695e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240511146119680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236510362108756 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46423064 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105215834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3637619114836 E³ÎtsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232872742993920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3650477662720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240511041136128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46423066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14043235 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46423066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240511146119680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105205841 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58782777 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42581801 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222903403491328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2919.047858661 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3650477662720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222903428395008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28538792 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105205841 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226675463230976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 trainer/global_step‚6199 _step‚501 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42582027 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58792768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚285387898×Ì ã&¤ÝM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28538789 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58792768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3772034835968 % _timestamp‚1.7843709776037695e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240511146119680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236510362108756 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46423064 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105215834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3637619114836 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240511041136128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46423066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14043235 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46423066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240511146119680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105205841 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58782777 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42581801 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222903403491328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2919.047858661 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3650477662720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222903428395008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28538792 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105205841 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226675463230976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 trainer/global_step‚6199 _step‚501 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42582027 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58792768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991õ75î½2¥- _wandbruntime‚2919  _runtime‚2919Mû6¦ÿ 2loss/FBt2/assignment_loss‚0.037015192433927306 trainer/global_step‚6199 /loss/FBt2/detection_loss‚0.5401935053684203 epoch‚7 0loss/FRt2/assignment_loss‚0.6712010958128505 1loss/SRqqt1/assignment_loss‚0.646758896443829 2loss/SRqqt2/assignment_loss‚0.5128680755095796 2loss/FBt1/assignment_loss‚0.005441063005734618 /loss/FRt1/detection_loss‚0.6161316534675542 /loss/FRt2/detection_loss‚1.4769053999895092 0loss/FRt1/assignment_loss‚1.6806248885381523 /loss/FBt1/detection_loss‚1.0226989768191495 $loss/total_loss‚8.70246093322196 1loss/SRqqt1/detection_loss‚0.8456310936722885 0loss/SRqqt2/detection_loss‚0.646991092160967 _runtime‚ 2919.04907133 _step‚502 % _timestamp‚1.7843709776055348e+09‰  §„ 0loss/FRt2/assignment_loss‚0.6712010958128505 1loss/SRqqt1/assignment_loss‚0.646758896443829 2loss/SRqqt2/assignment_loss‚0.5128680755095796 2loss/FBt1/assignment_loss‚0.005441063005734618 /loss/FRt1/detection_loss‚0.6161316534675542 /loss/FRt2/detection_loss‚1.4769053999895092 0loss/FRt1/assignment_loss‚1.6806248885381523 /loss/FBt1/detection_loss‚1.0226989768191495 $loss/total_loss‚8.70246093322196 1loss/SRqqt1/detection_loss‚0.8456310936722885 0loss/SRqqt2/detection_loss‚0.646991092160967 _runtime‚ 2919.04907133 _step‚502 % _timestamp‚1.7843709776055348e+09 2loss/FBt2/assignment_loss‚0.037015192433927306 trainer/global_step‚6199 /loss/FBt2/detection_loss‚0.5401935053684203 epoch‚7ög« œ¨‚(:‘ §®íÒ—÷©Š gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.573' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.514599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÞÑ©‚(:Æ §®íÒ–ì© disk.sda.out‚ 69.23828125$ disk.nvme0n1.out‚4892.7197265625! disk.dm-0.out‚4892.7197265625( proc.memory.availableMB‚ 130.86328125) disk./.usagePercent‚39.61466063395494 disk.dm-1.out‚ 69.23828125 cpu‚26.897523415081 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda1.out‚0 network.recv‚947240$ disk./.usageGB‚590.2870712280273 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3799.1640625 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 3799.1640625# proc.memory.rssMB‚ 2689.15234375 network.sent‚ 8.230375e+06 disk.rbd1.out‚7.1875 disk.nvme0n1.in‚ 3799.1640625 disk.sda2.out‚0 disk.sda.in‚0.109375& disk.nvme0c0n1.out‚4892.7197265625 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚99.20127391815186 disk.drbd2165.in‚0 disk.sda3.out‚ 69.23828125* proc.memory.percent‚16.413283348083496 disk.rbd0.out‚0 disk.sda3.in‚0.109375Ç8 œª‚(:‘ ¶®íÒ蘊‹ gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚143.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.799599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/²”ŸÑ«‚(:Æ ¶®íÒ´³™ network.recv‚952217 disk.sda.out‚ 69.50390625 disk.drbd2165.out‚0$ memory_percent‚99.20282363891602* proc.memory.percent‚16.413283348083496 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 3799.1640625 network.sent‚ 8.293104e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2689.15234375 disk.sda.in‚0.109375 disk.sda2.out‚0$ disk.nvme0n1.out‚4895.6689453125& proc.memory.availableMB‚ 130.609375 disk.rbd1.out‚7.1875 disk.dm-1.out‚ 69.50390625 cpu‚26.89416869999386) disk./.usagePercent‚39.61462120879467$ disk./.usageGB‚590.2864837646484" disk.nvme0c0n1.in‚ 3799.1640625& disk.nvme0c0n1.out‚4895.6689453125 disk.nvme0n1.in‚ 3799.1640625! disk.dm-0.out‚4895.6689453125 disk.sda3.out‚ 69.50390625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0.109375 proc.cpu.threads‚17 Qeâ2¬- _wandbruntime‚2940  _runtime‚2940MÄ&MÆ­À & lr-AdamW/pg1‚0.0008083789799739448 & lr-AdamW/pg2‚0.0008083789799739448 trainer/global_step‚6249 $ _timestamp‚1.784370999116878e+09 _runtime‚2940.560751141 _step‚5030Ž}´Ë®Å & lr-AdamW/pg2‚0.0008083789799739448 trainer/global_step‚6249 $ _timestamp‚1.784370999116878e+09 _runtime‚2940.560751141 _step‚503 & lr-AdamW/pg1‚0.0008083789799739448÷~&ü¨2¯- _wandbruntime‚2940  _runtime‚2940×3à)Ó'°ÍO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234659855685632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46776893 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14149309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242357514225152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242357514225152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42908050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚238325219321444 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46776893 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59239398 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224614067707904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59249375 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14149542 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238679088664576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3678320584192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42907814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242357409248768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3665449772902 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3678408521216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106026270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238679105703936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59249375 $ _timestamp‚1.784370999118048e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46776895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28758508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28758505 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228415027466240 trainer/global_step‚6249 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3678408521216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238325322497894 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238679088664576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242357409248768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234659872724992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3678320584192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚2940.562030305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224614092611584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106016291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59239398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 1937192966žÊq±ÒO trainer/global_step‚6249 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3678408521216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238325322497894 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238679088664576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242357409248768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234659872724992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3678320584192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚2940.562030305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224614092611584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106016291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59239398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234659855685632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46776893 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14149309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242357514225152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242357514225152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937020ð^üÕ4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42908050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚238325219321444 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46776893 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59239398 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224614067707904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59249375 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14149542 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238679088664576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3678320584192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42907814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242357409248768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3665449772902 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3678408521216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106026270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238679105703936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59249375 $ _timestamp‚1.784370999118048e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46776895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28758508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28758505 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228415027466240øN2²- _wandbruntime‚2940  _runtime‚2940mq÷gÞ&³ØM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46784116 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242395089572352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42914416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242395194555904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224649004126208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59258670 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106032793 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46784114 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228450531990528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2940.905142399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238716216668160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46784116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28762989 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224648979222528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234696327720960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238716199628800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46784114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843709994610653e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59248679 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242395194555904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106042786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238362361854864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59258670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3801527864320 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3678889943552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59248679 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0‚§ƒã&´ÝM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59258670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3801527864320 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3678889943552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59248679 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46784116 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242395089572352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42914416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242395194555904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224649004126208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59258670 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106032793 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46784114 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228450531990528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2940.905142399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238716216668160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46784116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28762989 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224648979222528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234696327720960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238716199628800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46784114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843709994610653e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59248679 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242395194555904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106042786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238362361854864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327ùzÏeT2µ- _wandbruntime‚2940  _runtime‚2940ìÂT¶€ _step‚506 .loss/FRt1/detection_loss‚0.621962617867693 trainer/global_step‚6249 _runtime‚2940.906198462 .loss/FBt1/detection_loss‚1.042073780298233 /loss/FBt2/detection_loss‚0.5959624761971264 epoch‚7 $ _timestamp‚1.784370999462659e+09 /loss/FRt1/assignment_loss‚1.738079992835136 /loss/FRt2/assignment_loss‚0.747423109366809 2loss/SRqqt1/assignment_loss‚0.6800315767320346 /loss/FRt2/detection_loss‚1.5231456420039238 1loss/SRqqt1/detection_loss‚0.8654562649671097 %loss/total_loss‚8.991699971489158 2loss/SRqqt2/assignment_loss‚0.4983828973196221 4loss/FBt1/assignment_loss‚2.3467502821006292e-05 2loss/FBt2/assignment_loss‚0.016278718420048896 1loss/SRqqt2/detection_loss‚0.6628794279786017!v¤ ·… .loss/FRt1/detection_loss‚0.621962617867693 trainer/global_step‚6249 _runtime‚2940.906198462 .loss/FBt1/detection_loss‚1.042073780298233 /loss/FBt2/detection_loss‚0.5959624761971264 epoch‚7 $ _timestamp‚1.784370999462659e+09 /loss/FRt1/assignment_loss‚1.738079992835136 /loss/FRt2/assignment_loss‚0.747423109366809 2loss/SRqqt1/assignment_loss‚0.6800315767320346 /loss/FRt2/detection_loss‚1.5231456420039238 1loss/SRqqt1/detection_loss‚0.8654562649671097 %loss/total_loss‚8.991699971489158 2loss/SRqqt2/assignment_loss‚0.4983828973196221 4loss/FBt1/assignment_loss‚2.3467502821006292e-05 2loss/FBt2/assignment_loss‚0.016278718420048896 1loss/SRqqt2/detection_loss‚0.6628794279786017 _step‚506ú¤QÁ±œ¸‚(:‘ Å®íÒΰ‹ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚179.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.919200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©xŒÑ¹‚(:Æ Å®íÒØù disk.dm-1.in‚0.109375 disk.sda.in‚0.109375 disk.sda2.out‚0* proc.memory.percent‚16.413283348083496$ disk./.usageGB‚590.2865905761719 disk.sda3.in‚0.109375 disk.rbd1.in‚0& disk.nvme0c0n1.out‚4902.1865234375 disk.rbd0.in‚0 cpu‚26.890968129829364 proc.cpu.threads‚17 network.recv‚955603) disk./.usagePercent‚39.61462837700562 disk.rbd1.out‚7.25 disk.sda1.in‚0 disk.dm-1.out‚ 69.66015625 disk.sda.out‚ 69.66015625 disk.sda1.out‚0 disk.sda3.out‚ 69.66015625 disk.sda2.in‚0# memory_percent‚99.2027997970581( proc.memory.availableMB‚ 130.61328125! disk.dm-0.out‚4902.1865234375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3799.1640625 disk.rbd0.out‚0 network.sent‚ 8.298062e+06 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2689.15234375 disk.dm-0.in‚ 3799.1640625" disk.nvme0c0n1.in‚ 3799.1640625$ disk.nvme0n1.out‚4902.1865234375ÀÚ2º- _wandbruntime‚2962  _runtime‚2962‚ÉñÇ»Á & lr-AdamW/pg1‚0.0008076591102071644 & lr-AdamW/pg2‚0.0008076591102071644 trainer/global_step‚6299 % _timestamp‚1.7843710210046108e+09 _runtime‚2962.448521925 _step‚507Ï‘ÅÌ¼Æ trainer/global_step‚6299 % _timestamp‚1.7843710210046108e+09 _runtime‚2962.448521925 _step‚507 & lr-AdamW/pg1‚0.0008076591102071644 & lr-AdamW/pg2‚0.0008076591102071644û;Ö°2½- _wandbruntime‚2962  _runtime‚2962ãÏO#/ ¾ÎO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3830357894144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3830181590016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230189825029120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47137945 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236483457452032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47137945 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240534653915136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47137943 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240534636875776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43240722 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 OIDeviceStatsMonitor.on_train_batch_start/inacê¶f)¥tive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59705300 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14258248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14258017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226359668342784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚2962.449768608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _step‚508 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28982705 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106853222 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236483474491392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106843243 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244241457684992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240177219067552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230190026236928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59705300 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240534636875776 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240534653915136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244241562661376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240177322244002 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244241562661376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3706820809216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28982708 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244241457684992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3706908746240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3693761615520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59715277 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3706908746240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43240956 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843710210057788e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47137943 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3706820809216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59715277 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327i¡‘Ù'¿ÓO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59715277 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3706908746240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43240956 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843710210057788e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47137943 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3706820809216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59715277 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3830357894144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3830181590016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230189825029120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47137945 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236483457452032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47137945 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240534653915136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47137943 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240534636875776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43240722 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59705300 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14258248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14258017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226359668342784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚2962.449768608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _step‚508 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28982705 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106853222 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236483474491392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106843243 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244241457684992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240177219067552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230190026236928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59705300 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240534636875776 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240534653915136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244241562661376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240177322244002 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244241562661376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3706820809216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28982708 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244241457684992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3706908746240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3693761615520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192üH;!2À- _wandbruntime‚2962  _runtime‚2962UþcÞ&ÁØM BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106859745 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240571747840000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59724572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240571764879360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43247461 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106869738 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47145166 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230225528479744 trainer/global_step‚6299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244279242992128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3830948622336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47145166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230225320987648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843710213476586e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236519929487360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3694415074252 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚2962.791733827 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47145164 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106859745 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240214361600972 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14260512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14260272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226394579857408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3707478112768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240571764879360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106869738 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694328937050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226394554953728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240214258424410 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3707478112768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28987189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3707390168576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43247704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236519946526720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240571747840000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244279138008576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28987192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3830766033920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚509 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244279242992128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59724572ÐãºËOÂÝM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244279242992128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59724572 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106859745 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240571747840000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59724572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240571764879360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43247461 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106869738 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47145166 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230225528479744 trainer/global_step‚6299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244279242992128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3830948622336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47145166 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230225320987648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843710213476586e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236519929487360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3694415074252 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚2962.791733827 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47145164 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106859745 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_¸Gqd”train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240214361600972 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14260512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14260272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226394579857408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3707478112768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240571764879360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106869738 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694328937050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226394554953728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240214258424410 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3707478112768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28987189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3707390168576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43247704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236519946526720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240571747840000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244279138008576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28987192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3830766033920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚509 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101ýDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43573576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242390202126336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107680174 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚6349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3859627989504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3735408971264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246125506121216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242029218813660 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238307059218432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238307076257792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107670195 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242390185086976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚2984.108732054 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47498993 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3722245732318 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242390202126336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29206905 % _timestamp‚1.7843710426647434e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29206908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228105219170304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47498993 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3735408971264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3722159595228 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246125506121216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47498995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246125611097600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47498995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43573352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14366668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14366447 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246125611097600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228105244073984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3735321034240ϹÙ'ÍÓO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60171202 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246125506121216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47498995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246125611097600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47498995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43573352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14366668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14366447 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246125611097600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228105244073984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3735321034240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60171202 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231964847159808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43573576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242390202126336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107680174 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚6349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3859627989504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3735408971264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246125506121216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242029218813660 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238307059218432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238307076257792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107670195 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242390185086976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚2984.108732054 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47498993 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3722245732318 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242390202126336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29206905 % _timestamp‚1.7843710426647434e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29206908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228105219170304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47498993 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3735408971264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3722159595228 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60190474 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14368558 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47506216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14368771 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3722813053960 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3735890393600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246163186444800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107686697 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47506216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3735978337792 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/alloczŽ/@ated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232000351347200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚513 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107696690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29211389 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242066258170518 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843710430032065e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246163291428352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238343531253760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43579947 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228140130684928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3735978337792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47506214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3735890393600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242427296051200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246163291428352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242427313090560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60180483 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3860394862080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3860220662272 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2984.447263654 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43580163 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232000550450688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242066361347080 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60180483 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3722726916758 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107686697 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47506214 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238343548293120âÚ˜ ã&ÐÝM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60190474 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14368558 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47506216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14368771 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3722813053960 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3735890393600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246163186444800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107686697 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47506216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3735978337792 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232000351347200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚513 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107696690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29211389 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242066258170518 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843710430032065e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246163291428352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238343531253760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43579947 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228140130684928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3735978337792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47506214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3735890393600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242427296051200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246163291428352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242427313090560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60180483 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3860394862080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3860220662272 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2984.447263654 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43580163 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232000550450688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242066361347080 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60180483 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3722726916758 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107686697 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47506214 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238343548293120øÑÊ32Ñ- _wandbruntime‚2984  _runtime‚2984\ƒD Òƒ trainer/global_step‚6349 2loss/SRqqt1/assignment_loss‚0.5580732279441292 1loss/SRqqt2/assignment_loss‚0.549365115401675 1loss/SRqqt2/detection_loss‚0.6077803659054296 _runtime‚2984.448745111 3loss/FBt1/assignment_loss‚0.0008991011503015146 _step‚514 0loss/FRt2/assignment_loss‚0.5948192893134223 1loss/FBt2/assignment_loss‚0.04661337441862941 epoch‚7 % _timestamp‚1.7843710430052016e+09 0loss/FRt1/assignment_loss‚1.7078381121266775 /loss/FRt1/detection_loss‚0.5340325097565607 /loss/FRt2/detection_loss‚1.4903123540845182 1loss/SRqqt1/detection_loss‚0.8883050811579032 0loss/FBt1/detection_loss‚0.34460000115047607 /loss/FBt2/detection_loss‚0.5519266834216459 %loss/total_loss‚7.874565215831369ã¼a?Óˆ 0loss/FRt2/assignment_loss‚0.5948192893134223 1loss/FBt2/assignment_loss‚0.04661337441862941 epoch‚7 % _timestamp‚1.7843710430052016e+09 0loss/FRt1/assignment_loss‚1.7078381121266775 /loss/FRt1/detection_loss‚0.5340325097565607 /loss/FRt2/detection_loss‚1.4903123540845182 1loss/SRqqt1/detection_loss‚0.8883050811579032 0loss/FBt1/detection_loss‚0.34460000115047607 /loss/FBt2/detection_loss‚0.5519266834216459 %loss/total_loss‚7.874565215831369 trainer/global_step‚6349 2loss/SRqqt1/assignment_loss‚0.5580732279441292 1loss/SRqqt2/assignment_loss‚0.549365115401675 1loss/SRqqt2/detection_loss‚0.6077803659054296 _runtime‚2984.448745111 3loss/FBt1/assignment_loss‚0.0008991011503015146 _step‚514‚¬:ËÊ‘Ô‚(:† ã®íÒ郺Š gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚168.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bhŠ<ÓÕ‚(:È ã®íÒ¤¦ú* disk./.usagePercent‚39.614638617306994$ disk./.usageGB‚590.2867431640625 disk.rbd1.in‚0! disk.nvme0n1.in‚ 3844.14453125* proc.memory.percent‚16.413283348083496 disk.drbd2165.out‚0 disk.rbd0.out‚0$ memory_percent‚99.20258522033691 cpu‚26.88465318769753 disk.drbd2165.in‚0 disk.dm-1.out‚ 70.29296875 disk.dm-0.out‚4914.173828125' proc.memory.availableMB‚ 130.6484375 disk.sda.out‚ 70.29296875 disk.rbd1.out‚7.3125 disk.sda2.out‚0% disk.nvme0c0n1.out‚4914.173828125 disk.sda1.out‚0 disk.dm-1.in‚0.109375 proc.cpu.threads‚17 network.recv‚965423 disk.sda1.in‚0# proc.memory.rssMB‚ 2689.15234375 disk.sda3.in‚0.109375# disk.nvme0n1.out‚4914.173828125 disk.dm-0.in‚ 3844.14453125 disk.sda3.out‚ 70.29296875 disk.sda.in‚0.109375# disk.nvme0c0n1.in‚ 3844.14453125 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 8.423671e+06³±ŽÖ‚(:ƒ ò®íÒª¾ðŠ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SAÂ>Øׂ(:Í ò®íÒòÙ network.recv‚970374 disk.dm-1.out‚ 70.55078125 disk.drbd2165.out‚0 network.sent‚ 8.486284e+06) disk./.usagePercent‚39.61467497037685 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.sda3.out‚ 70.55078125 disk.rbd1.out‚7.3125# disk.nvme0c0n1.in‚ 3844.14453125% disk.nvme0n1.out‚4918.67919921875" memory_percent‚99.202561378479( proc.memory.availableMB‚ 130.65234375# proc.memory.rssMB‚ 2689.15234375* proc.memory.percent‚16.413283348083496! disk.nvme0n1.in‚ 3844.14453125 cpu‚26.881606931282697 disk.sda2.out‚0" disk.dm-0.out‚4918.67919921875 disk.sda.in‚0.109375 disk.sda.out‚ 70.55078125' disk.nvme0c0n1.out‚4918.67919921875$ disk./.usageGB‚590.2872848510742 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3844.14453125 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0ÕóëV2Ø- _wandbruntime‚3006  _runtime‚3006À4Î&ÇÙÁ & lr-AdamW/pg1‚0.0008062009485228393 & lr-AdamW/pg2‚0.0008062009485228393 trainer/global_step‚6399 % _timestamp‚1.7843710645871692e+09 _runtime‚3006.030955001 _step‚515\ùÉÌÚÆ trainer/global_step‚6399 % _timestamp‚1.7843710645871692e+09 _runtime‚3006.030955001 _step‚515 & lr-AdamW/pg1‚0.0008062009485228393 & lr-AdamW/pg2‚0.0008062009485228393ƒ¡„r2Û- _wandbruntime‚3006  _runtime‚3006 µÓ'ÜÍO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47860043 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248009659533824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3763909196288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243881321736218 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60637104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248009554557440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763821259264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29431105 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763821259264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243881218559768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244245733298176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233739915020800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248009659533824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108507126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47860043 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233740114131456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248009554557440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244245733298176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14475345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240130678024192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229850794901504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3750643712026 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43906679 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 trainer/global_step‚6399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43906450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_star I*’A"t/inactive_split.small_pool.peak‚452 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248009659533824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3763909196288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243881321736218 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60637104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248009554557440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763821259264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29431105 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763821259264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243881218559768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244245733298176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233739915020800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248009659533824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108507126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47860043 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233740114131456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248009554557440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244245733298176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14475345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240130678024192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229850794901504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3750643712026 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43906679 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 trainer/global_step‚6399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43906450 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47860043 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60646385 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3764478562816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240167150059520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43913136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229885731319808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14477791 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47867264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244282861301760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248047234881024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244282861301760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229885706416128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3889708253696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47867264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47867266 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47867266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚517 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60646385 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29435589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248047339864576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108523642 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244282844262400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764390618624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43913383 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚3006.375014216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108523642 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3889882453504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243918257916626 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 % _timestamp‚1.7843710649308991e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244282844262400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3764478562816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679@™„èã&àÝM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248047234881024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244282861301760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229885706416128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3889708253696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47867264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47867266 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47867266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚517 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60646385 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29435589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248047339864576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108523642 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244282844262400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764390618624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43913383 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚3006.375014216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108523642 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3889882453504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243918257916626 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 % _timestamp‚1.7843710649308991e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244282844262400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3764478562816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60656376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚6399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3764390618624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233775414669824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14477547 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60646385 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3764478562816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240167150059520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43913136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229885731319808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14477791 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47867264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244282861301760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640…Óå 2á- _wandbruntime‚3006  _runtime‚3006ßšæwâ‚ .loss/FBt2/detection_loss‚0.398896598112835 0loss/FRt1/assignment_loss‚1.7059470759032995 2loss/SRqqt1/assignment_loss‚0.5955240356145654 3loss/FBt2/assignment_loss‚0.0007278124457808393 &loss/total_loss‚7.9378999225160065 epoch‚7 _step‚518 0loss/FRt2/assignment_loss‚0.7024843750855861 1loss/SRqqt1/detection_loss‚0.8185500817771039 /loss/FBt1/detection_loss‚0.4357758514583111 trainer/global_step‚6399 % _timestamp‚1.7843710649325426e+09 2loss/SRqqt2/assignment_loss‚0.5539838423350842 1loss/SRqqt2/detection_loss‚0.5928895256874551 _runtime‚3006.376084937 1loss/FBt1/assignment_loss‚0.03793738468875034 /loss/FRt1/detection_loss‚0.5181137458424127 .loss/FRt2/detection_loss‚1.577069593564822¨V-Π㇠0loss/FRt1/assignment_loss‚1.7059470759032995 2loss/SRqqt1/assignment_loss‚0.5955240356145654 3loss/FBt2/assignment_loss‚0.0007278124457808393 &loss/total_loss‚7.9378999225160065 epoch‚7 _step‚518 0loss/FRt2/assignment_loss‚0.7024843750855861 1loss/SRqqt1/detection_loss‚0.8185500817771039 /loss/FBt1/detection_loss‚0.4357758514583111 trainer/global_step‚6399 % _timestamp‚1.7843710649325426e+09 2loss/SRqqt2/assignment_loss‚0.5539838423350842 1loss/SRqqt2/detection_loss‚0.5928895256874551 _runtime‚3006.376084937 1loss/FBt1/assignment_loss‚0.03793738468875034 /loss/FRt1/detection_loss‚0.5181137458424127 .loss/FRt2/detection_loss‚1.577069593564822 .loss/FBt2/detection_loss‚0.398896598112835†ÝæArœä‚(:‘ ¯íÒ„Æ¡‹ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÝÐå‚(:Å ¯íÒà ýŒ! disk.dm-0.out‚4927.7705078125 disk.rbd0.out‚0) disk./.usagePercent‚39.61467266630905 disk.dm-0.in‚ 3844.14453125 disk.rbd1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚590.2872505187988 disk.sda3.in‚0.109375# disk.nvme0c0n1.in‚ 3844.14453125 disk.sda1.out‚0' proc.memory.availableMB‚ 129.6328125 network.recv‚974035 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚26.88498766314314 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 70.7890625$ memory_percent‚99.20878410339355( proc.memory.percent‚16.4201021194458& disk.nvme0c0n1.out‚4927.7705078125! disk.nvme0n1.in‚ 3844.14453125$ disk.nvme0n1.out‚4927.7705078125 disk.dm-1.in‚0.109375# proc.memory.rssMB‚ 2690.26953125 network.sent‚ 8.491277e+06 disk.sda3.out‚ 70.7890625 disk.sda.out‚ 70.7890625 disk.drbd2165.in‚0 disk.rbd1.out‚7.4375ª]*2æ- _wandbruntime‚3029  _runtime‚3029è‘šç£{ <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_4‚0.9579459428787231 particle/accuracy_1_of_3‚1 jet/accuracy_4_of_5‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4027777910232544 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7081412076950073 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899723529815674 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7465306520462036 @&Purity/2Fû•¾Ì:Rt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3893515169620514 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7650588750839233 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 / Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.56640625 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15194541215896606 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47914260625839233 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6703962087631226 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9838112592697144 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.587445855140686 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9025710821151733 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12343087792396545 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8469298481941223 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6597222685813904 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9881471991539001 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6719818711280823 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9852206707000732 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969446063041687 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006572360638529062 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2907826900482178 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933385252952576 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6676203012466431 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7476977109909058 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49860164523124695 particle/accuracy_1_of_1‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.994996190071106 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5840998291969299 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997687041759491 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08433561772108078 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980652332305908 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950268864631653 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4522731900215149 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7499222755432129 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011822698637843132 +particle/specificity‚0.9102897644042969 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997687041759491 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6649830937385559 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07489677518606186 ,jet/accuracy_4_of_4‚0.014682539738714695 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7206292748451233 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7048594355583191 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6654075384140015 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.034428227692842484 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6700699925422668 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.749833345413208 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030497660860419273 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8469298481941223 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8013709187507629 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08398254960775375 (loss/val_total_loss‚8.28948768613231 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 +jet/accuracy_3_of_3‚0.07498131692409515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 jet/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9402868151664734 particle/accuracy_2_of_2‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07485735416412354 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14529159665107727 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8136140704154968 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8761425018310547 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09854691475629807 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1274489015340805 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3085188567638397 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6670144200325012 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13623686134815216 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18333333730697632 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6917940378189087 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6970968246459961 .particle/accuracy_3_of_3‚0.999514102935791 +particle/sensitivity‚0.6375311613082886 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25056663155555725 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898993968963623 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939611554145813 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.746913731098175 *jet/accuracy_1_of_1‚0.3111049234867096 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7051120400428772 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6439343690872192 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7026875615119934 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11632312834262848 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7165379524230957 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15148471295833588 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7144240140914917 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7675427198410034 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20451748371124268 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1654585301876068 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_2_of_2‚0.14987172186374664 _runtime‚3029.082572736 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.648969292640686 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5020817518234253 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3931125998497009 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4655197560787201 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3919752836227417 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14287003874778748 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05954786017537117 (particle/accuracy‚0.8517135381698608 particle/accuracy_1_of_5‚1 epoch‚7 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15128597617149353 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.782389223575592 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8013709187507629 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8747158050537109 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9762944579124451 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.022123798727989197 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15661680698394775 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 $ _timestamp‚1.784371087637868e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15080603957176208 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925132989883423 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5120803117752075 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6397656798362732 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1791544258594513 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20843233168125153 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 *jet/accuracy_2_of_4‚0.7683232426643372 +jet/accuracy_3_of_4‚0.09330946207046509 'particle/f_score‚0.6486455798149109 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6470175981521606 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3588654100894928 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8459899425506592 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9187104105949402 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6700699925422668 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11452529579401016 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16576817631721497 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 trainer/global_step‚6407 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16467131674289703 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05954786017537117 *jet/accuracy_1_of_2‚0.7910189628601074 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6568613052368164 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996039867401123 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6165242195129395 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983720183372498 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13684000074863434 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9306016564369202 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1562705934047699 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6046038866043091 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1361532360315323 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15128597617149353 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7526297569274902 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7583613395690918 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_5‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6583523750305176 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7752135992050171 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 *jet/accuracy_2_of_3‚0.5570236444473267 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8230769038200378 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16721327602863312 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43939390778541565 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016802942380309105 _step‚519 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10487055778503418 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6704307794570923 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7827479243278503 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9584567546844482 jet/accuracy_5_of_5‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.223040372133255(ÔÙk®=è¨{ ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925132989883423 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5120803117752075 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6397656798362732 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1791544258594513 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20843233168125153 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 *jet/accuracy_2_of_4‚0.7683232426643372 +jet/accuracy_3_of_4‚0.09330946207046509 'particle/f_score‚0.6486455798149109 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6470175981521606 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3588654100894928 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8459899425506592 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9187104105949402 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6700699925422668 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11452529579401016 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16576817631721497 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 trainer/global_step‚6407 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16467131674289703 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05954786017537117 *jet/accuracy_1_of_2‚0.7910189628601074 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6568613052368164 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996039867401123 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6165242195129395 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983720183372498 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13684000074863434 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9306016564369202 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1562705934047699 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6046038866043091 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1361532360315323 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15128597617149353 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7526297569274902 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7583613395690918 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_5‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6583523750305176 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7752135992050171 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 *jet/accuracy_2_of_3‚0.5570236444473267 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8230769038200378 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16721327602863312 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43939390778541565 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016802942380309105 _step‚519 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10487055778503418 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6704307794570923 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7827479243278503 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9584567546844482 jet/accuracy_5_of_5‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.223040372133255 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_4‚0.9579459428787231 particle/accuracy_1_of_3‚1 jet/accuracy_4_of_5‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4027777910232544 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7081412076950073 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899723529815674 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7465306520462036 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3893515169620514 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7650588750839233 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 / Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.56640625 particle/accuracy_1_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15194541215896606 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47914260625839233 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6703962087631226 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9838112592697144 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.587445855140686 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9025710821151733 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12343087792396545 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8469298481941223 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6597222685813904 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9881471991539001 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6719818711280823 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9852206707000732 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969446063041687 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006572360638529062 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2907826900482178 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933385252952576 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6676203012466431 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7476977109909058 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49860164523124695 particle/accuracy_1_of_1‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.994996190071106 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5840998291969299 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997687041759491 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08433561772108078 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980652332305908 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950268864631653 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4522731900215149 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7499222755432129 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011822698637843132 +particle/specificity‚0.9102897644042969 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997687041759491 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6649830937385559 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07489677518606186 ,jet/accuracy_4_of_4‚0.014682539738714695 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7206292748451233 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7048594355583191 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6654075384140015 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.034428227692842484 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6700699925422668 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.749833345413208 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030497660860419273 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8469298481941223 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8013709187507629 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08398254960775375 (loss/val_total_loss‚8.28948768613231 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 +jet/accuracy_3_of_3‚0.07498131692409515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 jet/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9402868151664734 particle/accuracy_2_of_2‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07485735416412354 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14529159665107727 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8136140704154968 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8761425018310547 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09854691475629807 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1274489015340805 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3085188567638397 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6670144200325012 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13623686134815216 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18333333730697632 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6917940378189087 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6970968246459961 .particle/accuracy_3_of_3‚0.999514102935791 +particle/sensitivity‚0.6375311613082886 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25056663155555725 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898993968963623 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939611554145813 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.746913731098175 *jet/accuracy_1_of_1‚0.3111049234867096 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7051120400428772 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6439343690872192 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7026875615119934 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11632312834262848 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7165379524230957 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15148471295833588 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7144240140914917 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7675427198410034 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20451748371124268 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1654585301876068 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_2_of_2‚0.14987172186374664 _runtime‚3029.082572736 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.648969292640686 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5020817518234253 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3931125998497009 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4655197560787201 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3919752836227417 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14287003874778748 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05954786017537117 (particle/accuracy‚0.8517135381698608 particle/accuracy_1_of_5‚1 epoch‚7 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15128597617149353 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.782389223575592 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8013709187507629 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8747158050537109 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9762944579124451 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.022123798727989197 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15661680698394775 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 $ _timestamp‚1.784371087637868e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15080603957176208 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752‡oæ4½é‚(:… ¯íү̿Š gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0賞Ñê‚(:Æ ¯íÒ‰åŒ disk.sda1.in‚0 disk.dm-1.out‚ 71.06640625* disk./.usagePercent‚39.614690330828914 disk.rbd0.out‚0 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.sda.in‚0.109375# memory_percent‚98.9875078201294 network.sent‚ 8.554102e+06$ disk.nvme0n1.out‚4930.5732421875! disk.dm-0.out‚4930.5732421875 disk.sda.out‚ 71.06640625! disk.nvme0c0n1.in‚ 3957.515625 disk.sda2.out‚0 disk.sda3.out‚ 71.06640625( proc.memory.availableMB‚ 165.88671875# proc.memory.rssMB‚ 2725.43359375 network.recv‚979230$ disk./.usageGB‚590.2875137329102 disk.rbd0.in‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 3957.515625* proc.memory.percent‚16.634726524353027 cpu‚26.900026024803577 disk.sda1.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚4930.5732421875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 3957.515625 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚7.4375D˜ú‘ë‚(:† Ÿ¯íÒœЊ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚186.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tç72'ì‚(:¼ Ÿ¯íÒž¶ÊŒ network.recv‚984511 disk.sda2.in‚0" disk.dm-0.out‚5062.89794921875( proc.memory.availableMB‚ 198.48046875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚7.5' disk.nvme0c0n1.out‚5062.89794921875 disk.sda.out‚ 71.265625 disk.nß#÷D vme0n1.in‚ 3957.515625 network.sent‚ 8.611599e+06) disk./.usagePercent‚39.61467036224124 disk.rbd0.in‚0 disk.sda3.in‚0.109375! disk.nvme0c0n1.in‚ 3957.515625 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 71.265625 cpu‚26.896856614898955 disk.sda3.out‚ 71.265625 disk.drbd2165.in‚0% disk.nvme0n1.out‚5062.89794921875$ memory_percent‚98.78857135772705 disk.dm-0.in‚ 3957.515625 disk.sda.in‚0.109375 disk.drbd2165.out‚0$ disk./.usageGB‚590.2872161865234 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2692.71875) proc.memory.percent‚16.43505096435547P¯$s2í- _wandbruntime‚3048  _runtime‚3048Ž˜ÄYÇîÁ trainer/global_step‚6449 % _timestamp‚1.7843711068814101e+09 _runtime‚3048.326616811 _step‚520 & lr-AdamW/pg1‚0.0008054626799607139 & lr-AdamW/pg2‚0.0008054626799607139FfLÌïÆ _runtime‚3048.326616811 _step‚520 & lr-AdamW/pg1‚0.0008054626799607139 & lr-AdamW/pg2‚0.0008054626799607139 trainer/global_step‚6449 % _timestamp‚1.7843711068814101e+09ˆ¯‡×•2ð- _wandbruntime‚3048  _runtime‚3048ä"d:Ô'ñÎO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242911497664512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247062277902336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3808923278848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109702205 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250871201181184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚3048.326705749 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250871096204800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61256561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247062277902336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109712184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3935193695744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3795530293180 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44411673 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61256561 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48455623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246707027957692 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29759888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3808923278848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29759885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48455621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14651788 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236446077752832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232510884057088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247062260862976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61246584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250871096204800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232510908960768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250871201181184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236446283154944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚6449 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242911480625152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44411904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14652016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935374194176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143\NdÙ'òÓO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3808923278848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29759885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48455621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14651788 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236446077752832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232510884057088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247062260862976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61246584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250871096204800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232510908960768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250871201181184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236446283154944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚6449 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242911480625152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44411904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14652016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935374194176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242911497664512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247062277902336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3808923278848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109702205 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250871201181184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚3048.326705749 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250871096204800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61256561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247062277902336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109712184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3935193695744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3795530293180 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44411673 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61256561 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48455623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246707027957692 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29759888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327‰ÛðNß2ó- _wandbruntime‚3048  _runtime‚3048vVÃÝ&ô×M H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109728700 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3796097614822 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44418338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14654205 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48462844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247099371827200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3796011477620 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109718707 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246744067314662 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3809404701184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242947952660480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3809492645376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236481780855296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3935960379904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236481577557504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29764372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48462842 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232545795571712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3809404701184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3048.665518361 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109718707 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48462842 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61255865 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250908881511936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250908776528384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29764369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232545820475392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109728700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14653969 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 $ _timestamp‚1.784371107221364e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 trainer/global_step‚6449 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61255865 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61265856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44418577 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242947969699840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48462844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3935781985792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246743964138100 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚522 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61265856 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247099388866560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250908881511936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562P2L¬gõÜM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242947952660480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3809492645376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236481780855296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3935960379904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236481577557504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitÎäØ‘{"or.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29764372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48462842 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232545795571712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3809404701184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3048.665518361 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109718707 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48462842 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61255865 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250908881511936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250908776528384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29764369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232545820475392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109728700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14653969 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 $ _timestamp‚1.784371107221364e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 trainer/global_step‚6449 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61255865 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61265856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44418577 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242947969699840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48462844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3935781985792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246743964138100 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚522 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61265856 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247099388866560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250908881511936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109728700 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3796097614822 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44418338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14654205 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48462844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247099371827200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3796011477620 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109718707 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246744067314662 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3809404701184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231ŠGiå2ö- _wandbruntime‚3048  _runtime‚3048\/÷€ trainer/global_step‚6449 .loss/FRt2/detection_loss‚1.526927409497531 epoch‚8 % _timestamp‚1.7843711072230473e+09 _runtime‚3048.666605473 /loss/FRt1/detection_loss‚0.5901922888908925 /loss/FBt1/detection_loss‚0.7447337181307375 0loss/FRt1/assignment_loss‚1.6294660982981355 0loss/FBt1/assignment_loss‚0.0740157046917087 3loss/FBt2/assignment_loss‚0.0005912188710707025 1loss/SRqqt1/detection_loss‚0.8969234774660279 /loss/FBt2/detection_loss‚0.4137591369768703 %loss/total_loss‚8.607722913367539 _step‚523 0loss/FRt2/assignment_loss‚0.8893167473482744 1loss/SRqqt1/assignment_loss‚0.611488254804049 2loss/SRqqt2/assignment_loss‚0.5844867210358486 1loss/SRqqt2/detection_loss‚0.6458221373563929[ÂÞf ø… 0loss/FRt2/assignment_loss‚0.8893167473482744 1loss/SRqqt1/assignment_loss‚0.611488254804049 2loss/SRqqt2/assignment_loss‚0.5844867210358486 1loss/SRqqt2/detection_loss‚0.6458221373563929 trainer/global_step‚6449 .loss/FRt2/detection_loss‚1.526927409497531 epoch‚8 % _timestamp‚1.7843711072230473e+09 _runtime‚3048.666605473 /loss/FRt1/detection_loss‚0.5901922888908925 /loss/FBt1/detection_loss‚0.7447337181307375 0loss/FRt1/assignment_loss‚1.6294660982981355 0loss/FBt1/assignment_loss‚0.0740157046917087 3loss/FBt2/assignment_loss‚0.0005912188710707025 1loss/SRqqt1/detection_loss‚0.8969234774660279 /loss/FBt2/detection_loss‚0.4137591369768703 %loss/total_loss‚8.607722913367539 _step‚523‹|C?6‘ù‚(:† ®¯íÒïÒƒ‹ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚171.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%€]IÈú‚(:½ ®¯íÒÄïÓŒ network.sent‚ 8.61655e+06 disk.nvme0n1.in‚ 3957.515625 disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.rbd1.out‚7.5$ memory_percent‚98.78880977630615 cpu‚26.893631816899937) disk./.usagePercent‚39.61466805817343! disk.nvme0c0n1.in‚ 3957.515625 disk.sda2.out‚0 disk.dm-1.out‚ 71.62890625 disk.sda.out‚ 71.62890625$ disk.nvme0n1.out‚5066.3818359375 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 198.44140625 proc.cpu.threads‚17# disk./.usageGB‚590.287181854248 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3957.515625 disk.sda3.in‚0.109375 disk.sda3.out‚ 71.62890625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2692.71875) proc.memory.percent‚16.43505096435547 network.recv‚987772& disk.nvme0c0n1.out‚5066.3818359375 disk.dm-1.in‚0.109375! disk.dm-0.out‚5066.3818359375eš½2û- _wandbruntime‚3070  _runtime‚3070iH‘ƒÆüÀ _runtime‚3070.264261753 _step‚524 & lr-AdamW/pg1‚0.0008047183019857577 & lr-AdamW/pg2‚0.0008047183019857577 trainer/global_step‚6499 $ _timestamp‚1.784371128820497e+09HУáËýÅ & lr-AdamW/pg2‚0.0008047183019857577 trainer/global_step‚6499 $ _timestamp‚1.784371128820497e+09 _runtime‚3070.264261753 _step‚524 & lr-AdamW/pg1‚0.0008047183019857577Œù‹1b2þ- _wandbruntime‚3070  _runtime‚3070œ¡›Ô'ÿÎO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44745365 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29984085 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48816671 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248917826113536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3837335566848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61712486 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3823842135798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚248917826113536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚248558924527350 trainer/global_step‚6499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248917809074176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252755249617408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238221112076288 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234256459788288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110539136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 % _timestamp‚1.7843711288214786e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14761280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234256484691968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61722463 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚244735099430912 _runtime‚3070.268604411 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3837423503872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252755249617408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3964826494976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 _step‚525 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244735082391552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48816673 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252755144641024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248917809074176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48816671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252755144641024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29984088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238221311186944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ëÍ8Ì'€ÓO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 _step‚525 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244735082391552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48816673 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252755144641024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248917809074176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48816671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252755144641024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29984088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238221311186944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44745365 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29984085 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48816671 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248917826113536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3837335566848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61712486 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3823842135798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚248917826113536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚248558924527350 trainer/global_step‚6499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248917809074176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252755249617408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238221112076288 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234256459788288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110539136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 % _timestamp‚1.7843711288214786e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14761280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234256484691968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61722463 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚244735099430912 _runtime‚3070.268604411 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3837423503872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252755249617408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 39648 26494976ù¸t2- _wandbruntime‚3070  _runtime‚3070Äxf¦Þ&‚ØM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110555652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252792824964608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248954920038400 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48823892 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248954937077760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3837992870400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48823894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29988572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3824495594530 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44752062 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3965244538880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61721767 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110545659 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244771571466240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238256615841792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚6499 _runtime‚3070.604308727 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248595963884208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61731758 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3837904926208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3965416641536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29988569 _step‚526 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14763493 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3824409457328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244771554426880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14763731 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2}¹xã&ƒÝM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61721767 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110545659 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244771571466240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238256615841792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚6499 _runtime‚3070.604308727 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248595963884208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61731758 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3837904926208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3965416641536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29988569 _step‚526 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14763493 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3824409457328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244771554426880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14763731 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110555652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252792824964608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248954920038400 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48823892 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248954937077760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3837992870400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48823894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29988572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3824495594530 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44752062 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3965244538880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Že!ßË2„- _wandbruntime‚3070  _runtime‚3070:Õb©…‚ 0loss/FRt2/assignment_loss‚0.6532616266608238 2loss/SRqqt2/assignment_loss‚0.5031491737261516 .loss/FRt2/detection_loss‚1.543572516394779 1loss/SRqqt1/detection_loss‚1.0064852660382702 trainer/global_step‚6499 _runtime‚3070.605773262 0loss/FRt1/assignment_loss‚1.5989430259423427 1loss/SRqqt2/detection_loss‚0.6935894353309957 /loss/FBt1/detection_loss‚0.6320739560243156 epoch‚8 /loss/FRt1/detection_loss‚0.6363920597152709 /loss/FBt2/detection_loss‚0.6494736757838347 %loss/total_loss‚8.574440468803203 2loss/SRqqt1/assignment_loss‚0.6246593992432229 3loss/FBt1/assignment_loss‚0.0004308549496252725 1loss/FBt2/assignment_loss‚0.03240947899357181 % _timestamp‚1.7843711291622255e+09 _step‚527` †‡ 2loss/SRqqt1/assignment_loss‚0.6246593992432229 3loss/FBt1/assignment_loss‚0.0004308549496252725 1loss/FBt2/assignment_loss‚0.03240947899357181 % _timestamp‚1.7843711291622255e+09 _step‚527 0loss/FRt2/assignment_loss‚0.6532616266608238 2loss/SRqqt2/assignment_loss‚0.5031491737261516 .loss/FRt2/detection_loss‚1.543572516394779 1loss/SRqqt1/detection_loss‚1.0064852660382702 trainer/global_step‚6499 _runtime‚3070.605773262 0loss/FRt1/assignment_loss‚1.5989430259423427 1loss/SRqqt2/detection_loss‚0.6935894353309957 /loss/FBt1/detection_loss‚0.6320739560243156 epoch‚8 /loss/FRt1/detection_loss‚0.6363920597152709 /loss/FBt2/detection_loss‚0.6494736757838347 %loss/total_loss‚8.574440468803203»£T‘‡‚(:† ½¯íÒîèÀ gpu.0.gpu‚54.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u„+ƈ‚(:» ½¯íÒÇä‘ disk.drbd2165.in‚0$ memory_percent‚98.78876209259033) proc.memory.percent‚16.43507480621338& disk.nvme0c0n1.out‚5071.9443359375( proc.memory.availableMB‚ 198.44921875 network.recv‚993123$ disk./.usageGB‚590.2862510681152 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.dm-0.in‚ 3957.515625! disk.nvme0c0n1.in‚ 3957.515625# proc.memory.rssMB‚ 2692.72265625 disk.sda2.in‚0 disk.sda.out‚71.90625 disk.sda1.in‚0 disk.dm-1.out‚71.90625! disk.dm-0.out‚5071.9443359375 disk.nvme0n1.in‚ 3957.515625 disk.dm-1.in‚0.109375 disk.sda1.out‚0 disk.sda3.in‚0.109375 disk.rbd1.in‚0 cpu‚26.89063652346616 disk.rbd1.out‚7.5625$ disk.nvme0n1.out‚5071.9443359375 network.sent‚ 8.692271e+06) disk./.usagePercent‚39.61460559233508 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚71.90625 proc.cpu.threads‚17 disk.rbd0.out‚0¿Ó.瑉‚(:† ̯íÒÄŒ¨‹ gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™RΊ‚(:à ̯íÒ€´ disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4066.8046875 disk.sda2.out‚0 cpu‚26.88750503445085 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.rbd1.in‚0 disk.rbd1.out‚7.5625" disk.nvme0c0n1.in‚ 4066.8046875 network.recv‚998327) disk./.usagePercent‚39.61460533632754# disk./.usageGB‚590.286247253418# disk.nvme0n1.out‚5075.568359375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4066.8046875 disk.sda2.in‚0 disk.sda.out‚ 72.11328125 disk.dm-1.out‚ 72.11328125 disk.dm-0.out‚5075.568359375# proc.memory.rssMB‚ 2692.72265625$ memory_percent‚98.79024028778076( proc.memory.availableMB‚ 198.20703125 disk.sda.in‚0.109375) proc.memory.percent‚16.43507480621338 disk.sda3.out‚ 72.11328125% disk.nvme0c0n1.out‚5075.568359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0.109375 proc.cpu.threads‚17 network.sent‚ 8.742163e+06»Þ2‹- _wandbruntime‚3092  _runtime‚3092ð+-"ÅŒ¿ _runtime‚ 3092.10872784 _step‚528 & lr-AdamW/pg1‚0.0008039678265206951 & lr-AdamW/pg2‚0.0008039678265206951 trainer/global_step‚6549 $ _timestamp‚1.784371150664945e+09ÑÜzýÊÄ & lr-AdamW/pg2‚0.0008039678265206951 trainer/global_step‚6549 $ _timestamp‚1.784371150664945e+09 _runtime‚ 3092.10872784 _step‚528 & lr-AdamW/pg1‚0.0008039678265206951‚"uÆ2Ž- _wandbruntime‚3092  _runtime‚3092·&× ÍO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3865923728896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30208288 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62188365 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14871394 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3852240115506 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚529 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3865923728896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111366088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254639193077248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49177721 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45079682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246558684157952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14871153 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254639298053632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 % _timestamp‚1.7843711506659362e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49177723 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246558701197312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62188365 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62178388 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3852326252596 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3865835791872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250773374324736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3865835791872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3994284678144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250411027449908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250773374324736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_g\3 pool.freed‚ 3994106276864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45079438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254639193077248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239996141796352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236002060423168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236002035519488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250773357285376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250410924273458 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960“tØ'ÒO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45079438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254639193077248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239996141796352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236002060423168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236002035519488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250773357285376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250410924273458 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3865923728896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30208288 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62188365 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14871394 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3852240115506 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚529 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3865923728896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111366088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254639193077248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49177721 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45079682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246558684157952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14871153 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254639298053632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 % _timestamp‚1.7843711506659362e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49177723 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246558701197312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62188365 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62178388 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3852326252596 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3865835791872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250773374324736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3865835791872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3994284678144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250411027449908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250773374324736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3994106276864‘RGå2‘- _wandbruntime‚3092  _runtime‚3092àþ­Ü&’ÖM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49184944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14873269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236036947034112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30212769 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3852893574238 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49184942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246595156193280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250810485288960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3866405151232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3852807437036 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111382604 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111382604 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250810468249600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111372611 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _step‚530 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6549 $ _timestamp‚1.784371151010069e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49184944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3994696760320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250448066806878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45086038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚254676873400832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62187669 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3994870960128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62187669 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250810468249600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240031643794432 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250810485288960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3866405151232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3852807437036 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111382604 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111382604 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250810468249600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111372611 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _step‚530 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6549 $ _timestamp‚1.784371151010069e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49184944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3994696760320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250448066806878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45086038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚254676873400832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62187669 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3994870960128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62187669 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250810468249600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240031643794432 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3852893574238’†Ëì:2”- _wandbruntime‚3092  _runtime‚3092Òø­£• 0loss/FRt1/assignment_loss‚1.7167639303344682 /loss/FRt1/detection_loss‚0.5987886007038231 /loss/FRt2/detection_loss‚1.5559567444452747 0loss/=ˆ0dSRqqt2/detection_loss‚0.605024119380199 0loss/FRt2/assignment_loss‚0.5724051247781782 2loss/SRqqt2/assignment_loss‚0.5489099480791435 3loss/FBt1/assignment_loss‚4.060546121120723e-06 /loss/FBt1/detection_loss‚0.8631046918453649 /loss/FBt2/detection_loss‚0.3658850773050725 epoch‚8 $ _timestamp‚1.784371151011645e+09 2loss/FBt2/assignment_loss‚0.008300440853383441 %loss/total_loss‚8.179668426772732 trainer/global_step‚6549 _step‚531 2loss/SRqqt1/assignment_loss‚0.5613306705601284 0loss/SRqqt1/detection_loss‚0.783195017941575 _runtime‚3092.455184833Íbç" –† /loss/FRt1/detection_loss‚0.5987886007038231 /loss/FRt2/detection_loss‚1.5559567444452747 0loss/SRqqt2/detection_loss‚0.605024119380199 0loss/FRt2/assignment_loss‚0.5724051247781782 2loss/SRqqt2/assignment_loss‚0.5489099480791435 3loss/FBt1/assignment_loss‚4.060546121120723e-06 /loss/FBt1/detection_loss‚0.8631046918453649 /loss/FBt2/detection_loss‚0.3658850773050725 epoch‚8 $ _timestamp‚1.784371151011645e+09 2loss/FBt2/assignment_loss‚0.008300440853383441 %loss/total_loss‚8.179668426772732 trainer/global_step‚6549 _step‚531 2loss/SRqqt1/assignment_loss‚0.5613306705601284 0loss/SRqqt1/detection_loss‚0.783195017941575 _runtime‚3092.455184833 0loss/FRt1/assignment_loss‚1.7167639303344682“K0èq‘—‚(:† Û¯íÒµÝôŠ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00€¦ Ç˜‚(:¼ Û¯íÒ©éƒ disk.sda3.in‚0.109375$ disk./.usageGB‚590.2866706848145 disk.rbd1.out‚7.625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.sda.out‚72.28125' proc.memory.availableMB‚ 198.1953125* disk./.usagePercent‚39.614633753163844 disk.dm-1.out‚72.28125 disk.dm-0.out‚ 5082.61328125 network.sent‚ 8.747058e+06 disk.nvme0n1.in‚ 4066.8046875 disk.sda1.out‚0" disk.nvme0n1.out‚ 5082.61328125 disk.sda2.in‚0$ memory_percent‚98.79031181335449# proc.memory.rssMB‚ 2692.72265625 proc.cpu.threads‚17 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 5082.61328125 disk.rbd0.in‚0 disk.dm-0.in‚ 4066.8046875 disk.sda3.out‚72.28125 network.recv‚ 1.001946e+06 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 4066.8046875 disk.sda1.in‚0 disk.dm-1.in‚0.109375) proc.memory.percent‚16.43507480621338 cpu‚26.8843796009973 disk.drbd2165.out‚0mŠ!Ò2™- _wandbruntime‚3114  _runtime‚3114PPµÇšÁ _step‚532 & lr-AdamW/pg1‚0.0008032112655859139 & lr-AdamW/pg2‚0.0008032112655859139 trainer/global_step‚6599 % _timestamp‚1.7843711726393275e+09 _runtime‚3114.083137538{¶Ì›Æ % _timestamp‚1.7843711726393275e+09 _runtime‚3114.083137538 _step‚532 & lr-AdamW/pg1‚0.0008032112655859139 & lr-AdamW/pg2‚0.0008032112655859139 trainer/global_step‚6599”ñŠ 2œ- _wandbruntime‚3114  _runtime‚3114[ðÔ'ÎO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62644290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112193040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3894336016896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45413662 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45413899 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30432488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3880724232304 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49538771 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112183061 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62654267 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚256523346489856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _step‚533 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256523346489856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4023765613056 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252628905496576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241771196365312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14981411 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248382285924352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6599 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241771401767424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚252262924019566 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256523241513472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49538773 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252263027196016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3894423953920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3894336016896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252628905496576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248382302963712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252628922535936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49538773 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49538771 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3880638095214 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252628905496576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248382302963712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252628922535936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49538773 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49538771 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3880638095214 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _step‚533 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256523346489856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4023765613056 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252628905496576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241771196365312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14981411 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248382285924352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6599 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241771401767424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚252262924019566 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256523241513472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0•…dG2Ÿ- _wandbruntime‚3114  _runtime‚31141]% # ØM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252299963376424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62653571 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241806698426880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49545994 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237782522765312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252666016460800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚256561026820608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252666033500160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3894993320448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252666033500160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62663562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3114.426921699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112209556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30436969 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30436972 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237782547668992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45420293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 trainer/global_step‚6599 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49545992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14983561 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252666016460800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3894905376256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45420533 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚256561026820608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3894993320448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3894905376256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4024175661568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248418774999040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3881291553946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚534 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112199563 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62653571 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4024349861376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14983324 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49545994 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚°2''Ð 199103488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112209556 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252666033500160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3894993320448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252666033500160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62663562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3114.426921699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112209556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30436969 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30436972 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237782547668992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45420293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 trainer/global_step‚6599 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49545992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14983561 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252666016460800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3894905376256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45420533 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚256561026820608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3894993320448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3894905376256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4024175661568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248418774999040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3881291553946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚534 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112199563 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62653571 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4024349861376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14983324 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49545994 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112209556 F ¤‡ 0loss/FRt1/assignment_loss‚1.7397959245575798 0loss/FRt2/assignment_loss‚0.9066929759250747 2loss/FBt2/assignment_loss‚0.007283456065436401 %loss/total_loss‚8.422245992182454 epoch‚8 $ _timestamp‚1.784371172984464e+09 4loss/FBt1/assignment_loss‚0.00035439558943931173 /loss/FRt1/detection_loss‚0.5714033967876266 .loss/FRt2/detection_loss‚1.575532153541981 _runtime‚3114.428003873 _step‚535 2loss/SRqqt1/assignment_loss‚0.5322268894689184 2loss/SRqqt2/assignment_loss‚0.6241370237600347 /loss/SRqqt1/detection_loss‚0.78983286366689 1loss/SRqqt2/detection_loss‚0.6065861186964553 /loss/FBt1/detection_loss‚0.6493693594575712 0loss/FBt2/detection_loss‚0.41903143466544757 trainer/global_step‚6599— ©‘¥‚(:† ê¯íÒ—À¹Š gpu.0.gpu‚55.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚172.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ –צ‚(:Ì ê¯íÒΟ¢Ž$ disk./.usageGB‚590.2867889404297 disk.sda3.in‚0.109375 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚98.79207611083984# proc.memory.rssMB‚ 2692.72265625 disk.drbd2165.in‚0 disk.rbd1.out‚7.625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.nvme0n1.in‚ 4066.8046875 disk.rbd0.out‚0) disk./.usagePercent‚39.61464168939741" disk.dm-0.out‚5086.47216796875" disk.nvme0c0n1.in‚ 4066.8046875 cpu‚26.88125527438575 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 72.60546875 disk.dm-0.in‚ 4066.8046875 disk.sda3.out‚ 72.60546875% disk.nvme0n1.out‚5086.47216796875 network.sent‚ 8.810134e+06 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0) proc.memory.percent‚16.43507480621338 proc.cpu.threads‚17 network.recv‚ 1.006811e+06 disk.sda.out‚ 72.60546875' disk.nvme0c0n1.out‚5086.47216796875% proc.memory.availableMB‚ 197.90625À¦7œ§‚(:‘ ù¯íÒ½ÁêŠ gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&9´ˆÙ¨‚(:Î ù¯íÒ˜òŒ disk.sda2.in‚0 disk.dm-1.in‚0.109375" disk.dm-0.out‚5091.52099609375 network.sent‚ 8.872719e+06 disk.rbd1.in‚0$ memory_percent‚98.79553318023682 disk.sda.in‚0.109375 disk.nvme0n1.in‚ 4066.8046875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4066.8046875) proc.memory.percent‚16.43507480621338 proc.cpu.threads‚17) disk./.usagePercent‚39.61465500178918 disk.sda3.in‚0.109375 disk.rbd0.in‚0 network.recv‚ 1.012135e+06 disk.rbd1.out‚7.6875' disk.nvme0c0n1.out‚5091.52099609375 disk.sda.out‚ 72.7265625 disk.sda2.out‚0( proc.memory.availableMB‚ 197.33984375# proc.memory.rssMB‚ 2692.72265625$ disk./.usageGB‚590.2869873046875 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 4066.8046875% disk.nvme0n1.out‚5091.52099609375 disk.sda1.in‚0 disk.dm-1.out‚ 72.7265625 cpu‚26.878206476210888 disk.sda3.out‚ 72.7265625 disk.drbd2165.in‚0„€d2©- _wandbruntime‚3135  _runtime‚3135H(€1ê½ % lr-AdamW/pg2‚0.000802448631299273 trainer/global_step‚6649 $ _timestamp‚1.784371194398896e+09 _runtime‚ 3135.84270566 _step‚536 % lr-AdamW/pg1‚0.000802448631299273ËtõÈ«Â % lr-AdamW/pg2‚0.000802448631299273 trainer/global_step‚6649 $ _timestamp‚1.784371194398896e+09 _runtime‚ 3135.84270566 _step‚536 % lr-AdamW/pg1‚0.000802448631299273˜©»2¬- _wandbruntime‚3135  _runtime‚3135Óh„Ô'­ÎO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49899823 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3922836241920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113019992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254484470747136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30656688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15091476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239493186981888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63110192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258407289949696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49899821 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239493211885568 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250205904730112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4053020412416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843711943999782e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258407289949696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30656685 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250205887690752 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3922924178944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63120169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254484453707776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243546410699264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254115026942124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63110192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258407394926080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254484470747136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚537 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63120169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15091705 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4053198813696 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254484453707776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3922836241920 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977íÉÈý®ÓO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243546207394304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113010013 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254114923765674 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258407289949696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49899821 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239493211885568 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250205904730112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4053020412416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 % _timestamp‚1.7843711943999782e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258407289949696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30656685 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250205887690752 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3922924178944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63120169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254484453707776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243546410699264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254115026942124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63110192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258407394926080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254484470747136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _step‚537 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63120169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15091705 cQDeviceStatsMonitor.on_train_batch)ïrAÜ _start/inactive_split_bytes.small_pool.allocated‚ 4053198813696 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254484453707776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3922836241920 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49899823 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3922836241920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113019992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254484470747136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30656688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15091476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239493186981888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63110192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45754820 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49907044 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250242359726080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113026515 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45755054 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254521581711360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3923405601280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243581915213312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚538 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 % _timestamp‚1.7843711947380583e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258445075256832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254521564672000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49907042 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 193074176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113026515 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254152066299094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49907042 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113036508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113036508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49907044 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚250242376765440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚254521581711360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243581697235456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3909689533654 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4053791813120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254521564672000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚258445075256832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30661169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15093651 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3923493545472 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3136.182155434 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63129464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4053598738944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30661172 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÚR~ã&±ÝM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49907042 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113036508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113036508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49907044 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚250242376765440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚254521581711360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243581697235456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3909689533654 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4053791813120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254521564672000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚258445075256832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30661169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15093651 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3923493545472 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3136.182155434 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63129464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4053598738944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30661172 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45754820 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49907044 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250242359726080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113026515 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45755054 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254521581711360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3923405601280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243581915213312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚538 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 % _timestamp‚1.7843711947380583e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258445075256832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254521564672000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49907042 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 193074176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113026515 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254152066299094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0šãZc!2²- _wandbruntime‚3136  _runtime‚3136*¦´Ë³ _step‚539 0loss/FRt2/assignment_loss‚0.7252030527972161 2loss/FBt2/assignment_loss‚0.004387932580093754 /loss/FBt2/detection_loss‚0.5391377195747654 % _timestamp‚1.7843711947397065e+09 3loss/SRqqt1/assignment_loss‚0.47361819518444476 2loss/SRqqt2/assignment_loss‚0.5819690512693152 2loss/FBt1/assignment_loss‚0.005418984467679308 trainer/global_step‚6649 _runtime‚3136.183242508 1loss/SRqqt2/detection_loss‚0.6392701234887795 $loss/total_loss‚8.19168584648037 epoch‚8 0loss/FRt1/assignment_loss‚1.6496295726847374 .loss/FRt1/detection_loss‚0.552461289296711 /loss/FRt2/detection_loss‚1.5683015382432557 0loss/SRqqt1/detection_loss‚0.905277779099444 /loss/FBt1/detection_loss‚0.5470106077939272å<Ü ´† 1loss/SRqqt2/detection_loss‚0.6392701234887795 $loss/total_loss‚8.19168584648037 epoch‚8 0loss/FRt1/assignment_loss‚1.6496295726847374 .loss/FRt1/detection_loss‚0.552461289296711 /loss/FRt2/detection_loss‚1.5683015382432557 0loss/SRqqt1/detection_loss‚0.905277779099444 /loss/FBt1/detection_loss‚0.5470106077939272 _step‚539 0loss/FRt2/assignment_loss‚0.7252030527972161 2loss/FBt2/assignment_loss‚0.004387932580093754 /loss/FBt2/detection_loss‚0.5391377195747654 % _timestamp‚1.7843711947397065e+09 3loss/SRqqt1/assignment_loss‚0.47361819518444476 2loss/SRqqt2/assignment_loss‚0.5819690512693152 2loss/FBt1/assignment_loss‚0.005418984467679308 trainer/global_step‚6649 _runtime‚3136.183242508›@(Ÿœµ‚(:‘ ˆ°íÒ¥ÕÅŠ gpu.0.gpu‚15.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,¿þ-Ú¶‚(:Ï ˆ°íÒúѯ disk.rbd0.in‚0 disk.dm-0.in‚ 4118.23828125 disk.sda3.in‚0.109375 disk.sda3.out‚ 73.0546875 disk.sda.in‚0.109375 disk.drbd2165.out‚0 network.recv‚ 1.015447e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 73.0546875 disk.rbd1.out‚7.6875$ memory_percent‚99.18923377990723 disk.rbd0.out‚0 disk.dm-1.out‚ 73.0546875$ disk./.usageGB‚590.2876014709473' disk.nvme0c0n1.out‚5094.21240234375 disk.sda1.in‚0 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0' proc.memory.availableMB‚ 132.8359375( disk./.usagePercent‚39.6146962190022# disk.nvme0c0n1.in‚ 4118.23828125) proc.memory.percent‚16.43507480621338 cpu‚26.875232652464486 network.sent‚ 8.877669e+06 disk.sda2.out‚0" disk.dm-0.out‚5094.21240234375! disk.nvme0n1.in‚ 4118.23828125% disk.nvme0n1.out‚5094.21240234375# proc.memory.rssMB‚ 2692.72265625ïjÏL2·- _wandbruntime‚3157  _runtime‚3157qÛƸÀ & lr-AdamW/pg1‚0.0008016799358759086 & lr-AdamW/pg2‚0.0008016799358759086 trainer/global_step‚6699 % _timestamp‚1.7843712164067917e+09 _runtime‚ 3157.85164172 _step‚540ì=%@˹Š% _timestamp‚1.7843712164067917e+09 _runtime‚ 3157.85164172 _step‚540 & lr-AdamW/pg1‚0.0008016799358759086 & lr-AdamW/pg2‚0.0008016799358759086 trainer/global_step‚6699œmžƒ22º- _wandbruntime‚3157  _runtime‚3157g!­ˆ»ÍO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256340018958336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113836965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245321421167616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 _runtime‚3157.851735787 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260291443362304 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63586071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113846944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30880888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260291338385920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256340001918976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3937434054630 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46082630 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63576094 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚209 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255966923511782 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4082633550848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4082459343872 _step‚541 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3951336466944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3937520191720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_poolžÊœš´.current‚87937024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46082421 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241238762713088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260291338385920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245321222056960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63586071 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50260873 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252029506496512 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256340001918976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50260871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚260291443362304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3951336466944 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252029489457152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256340018958336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15201536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15201742 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3951424403968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50260873 trainer/global_step‚6699(íÝ+Ø'¼ÒO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260291338385920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256340001918976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3937434054630 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46082630 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63576094 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚209 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255966923511782 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4082633550848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4082459343872 _step‚541 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3951336466944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3937520191720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46082421 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241238762713088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260291338385920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245321222056960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63586071 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50260873 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252029506496512 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256340001918976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50260871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚260291443362304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3951336466944 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252029489457152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256340018958336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15201536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15201742 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3951424403968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50260873 trainer/global_step‚6699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256340018958336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113836965 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245321421167616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 _runtime‚3157.851735787 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260291443362304 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63586071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113846944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30880888&2ÉÄ2½- _wandbruntime‚3158  _runtime‚3158Q€‘Þ&¾ØM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260329123693056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4083049792512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63585375 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3951993770496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15203894 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113863460 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50268094 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63595366 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30885369 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚3158.191326837 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260329018709504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113863460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256377129922560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15203672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252065961492480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30885372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245356923123712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50268092 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245356724020224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚256004066045202 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113853467 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63585375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3938087513362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3951905826304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63595366 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46089266 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256003962868640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50268094 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚542 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256377112883200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50268092 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚256377129922560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3951993770496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252065978531840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3951905826304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46089041 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0{Ð*kA¿ÝM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260329018709504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113863460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256377129922560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15203672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252065961492480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30885372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245356923123712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50268092 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245356724020224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚256004066045202 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113853467 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63585375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3938087513362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3951905826304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63595366 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46089266 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256003962868640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50268094 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚542 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256377112883200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50268092 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚256377129922560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3951993770496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252065978531840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3951905826304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46089041 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260329123693056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4083049792512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63585375 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3951993770496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15203894 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZMDeviceStatsMonitor.on¥àzr¢_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113863460 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50268094 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63595366 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30885369 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚3158.191326837žUßXÇ2À- _wandbruntime‚3158  _runtime‚3158Ëã‹Á‚ 0loss/FRt2/assignment_loss‚0.7390318661186825 2loss/SRqqt1/assignment_loss‚0.5894155866643526 2loss/FBt1/assignment_loss‚0.010080543721139748 /loss/FRt1/detection_loss‚0.5390284618435154 %loss/total_loss‚8.475496698748003 epoch‚8 % _timestamp‚1.7843712167489524e+09 _step‚543 0loss/FRt1/assignment_loss‚1.7240883212436662 2loss/SRqqt2/assignment_loss‚0.5249958952820613 2loss/FBt2/assignment_loss‚0.008393507860612404 1loss/SRqqt2/detection_loss‚0.6915442950814887 /loss/FBt1/detection_loss‚0.8351507723331452 /loss/FBt2/detection_loss‚0.5262360688800422 trainer/global_step‚6699 /loss/FRt2/detection_loss‚1.5289795956671683 1loss/SRqqt1/detection_loss‚0.7585517840521288 _runtime‚ 3158.19249406?iNr ‡ 0loss/FRt2/assignment_loss‚0.7390318661186825 2loss/SRqqt1/assignment_loss‚0.5894155866643526 2loss/FBt1/assignment_loss‚0.010080543721139748 /loss/FRt1/detection_loss‚0.5390284618435154 %loss/total_loss‚8.475496698748003 epoch‚8 % _timestamp‚1.7843712167489524e+09 _step‚543 0loss/FRt1/assignment_loss‚1.7240883212436662 2loss/SRqqt2/assignment_loss‚0.5249958952820613 2loss/FBt2/assignment_loss‚0.008393507860612404 1loss/SRqqt2/detection_loss‚0.6915442950814887 /loss/FBt1/detection_loss‚0.8351507723331452 /loss/FBt2/detection_loss‚0.5262360688800422 trainer/global_step‚6699 /loss/FRt2/detection_loss‚1.5289795956671683 1loss/SRqqt1/detection_loss‚0.7585517840521288 _runtime‚ 3158.19249406Ÿý?;œÃ‚(:‘ —°íÒØã– gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.984' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.796800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿå?FÏÄ‚(:Ä —°íÒÕ¹çŽ network.sent‚ 8.940609e+06 network.recv‚ 1.020908e+06) disk./.usagePercent‚39.61475356468987$ disk./.usageGB‚590.2884559631348 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.109375" disk.dm-0.out‚5099.69287109375 disk.sda3.in‚0.109375 disk.sda3.out‚73.21875 disk.sda.in‚0.109375 disk.sda1.out‚0 disk.dm-1.out‚73.21875# proc.memory.rssMB‚ 2692.72265625 disk.sda.out‚73.21875 disk.rbd1.out‚7.75# disk.nvme0c0n1.in‚ 4166.45703125! disk.nvme0n1.in‚ 4166.45703125 disk.rbd0.in‚0% disk.nvme0n1.out‚5099.69287109375$ memory_percent‚99.18899536132812) proc.memory.percent‚16.43507480621338 disk.sda1.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚5099.69287109375 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4166.45703125# proc.memory.availableMB‚132.875 cpu‚26.872348342155025 proc.cpu.threads‚17æ.aÜ2Å- _wandbruntime‚3179  _runtime‚3179l¥(dÇÆÁ _runtime‚3179.715630446 _step‚544 & lr-AdamW/pg1‚0.0008009051916280381 & lr-AdamW/pg2‚0.0008009051916280381 trainer/global_step‚6749 % _timestamp‚1.7843712382717304e+09+¶bÌÇÆ _step‚544 & lr-AdamW/pg1‚0.0008009051916280381 & lr-AdamW/pg2‚0.0008009051916280381 trainer/global_step‚6749 % _timestamp‚1.7843712382717304e+09 _runtime‚3179.715630446 dÚ-¢2È- _wandbruntime‚3179  _runtime‚3179r1ÛôÓ'ÉÍO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31105085 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚242984338444288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3979924628992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64041996 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3979836691968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50621921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15312300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262175386822144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247096489737216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46417148 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4111952182272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262175386822144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3979924628992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚242984363347968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253853091223552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64051973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64051973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4112126389248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50621923 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114663917 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258195567169536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253853108262912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3965918171428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚6749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262175491798528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258195567169536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247096290626560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257818923257890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50621923 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3979836691968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15312063 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50621921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262175491798528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114673896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31105088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114663917 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258195567169536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253853108262912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3965918171428 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚6749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262175491798528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258195567169536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247096290626560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257818923257890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50621923 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3979836691968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15312063 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50621921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262175491798528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114673896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31105088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262175386822144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247096489737216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46417148 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4111952182272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262175386822144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3979924628992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚242984363347968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253853091223552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64051973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64051973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4112126389248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¡âÅXG›Ë‚(: ¦°íÒÂÜèŠ gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚182.63' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.525999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸéLÙÌ‚(:Î ¦°íÒ¥§˜ network.sent‚ 9.003421e+06 disk.dm-1.in‚0.109375) proc.memory.percent‚16.43507480621338 cpu‚26.869466111458056% proc.memory.availableMB‚ 132.65625% disk.nvme0n1.out‚5103.12451171875 disk.dm-1.out‚ 73.66796875# memory_percent‚99.1903305053711# proc.memory.rssMB‚ 2692.72265625) disk./.usagePercent‚39.61476662107412' disk.nvme0c0n1.out‚5103.12451171875 disk.rbd0.out‚0 disk.sda2.out‚0" disk.dm-0.out‚5103.12451171875 disk.sda3.in‚0.109375 disk.sda2.in‚0 network.recv‚ 1.025852e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.109375 disk.rbd1.out‚7.75! disk.nvme0n1.in‚ 4166.45703125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4166.45703125$ disk./.usageGB‚590.2886505126953 disk.drbd2165.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4166.45703125 disk.sda3.out‚ 73.66796875 disk.sda.out‚ 73.66796875 proc.cpu.threads‚17I…k72Í- _wandbruntime‚3180  _runtime‚3180=ËdôÎØM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243019274862592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253889580298240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3980493995520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64051277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15314533 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247131794570240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112714616832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258232678133760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262213067145728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50629144 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257856065791310 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3980406051328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843712386133687e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262213172129280 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50629142 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46423860 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31109572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15314291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114690412 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31109569 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStats~djyêMonitor.on_train_batch_end/requested_bytes.all.freed‚257855962614748 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3966399355868 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3980406051328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114680419 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50629142 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966485493070 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114690412 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64061268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64061268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114680419 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50629142 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966485493070 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114690412 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64061268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64061268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64051277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15314533 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247131794570240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112714616832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258232678133760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262213067145728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50629144 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257856065791310 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3980406051328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7843712386133687e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262213172129280 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50629142 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46423860 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31109572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15314291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114690412 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31109569 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257855962614748 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3966399355868¢¶­˜Ñ2Ð- _wandbruntime‚3180  _runtime‚3180v¦cÑ€ -loss/FRt1/detection_loss‚0.58379699519682 epoch‚8 trainer/global_step‚6749 0loss/FRt1/assignment_loss‚1.6758065719837971 /loss/FRt2/detection_loss‚1.6352610768905531 0loss/SRqqt1/detection_loss‚0.887801548888363 1loss/SRqqt2/detection_loss‚0.7084457473183045 /loss/FBt1/detection_loss‚0.7301587419850486 .loss/FBt2/detection_loss‚0.566276379479942 $loss/total_loss‚8.71680473513026 _step‚547 0loss/FRt2/assignment_loss‚0.7852699497231731 3loss/SRqqt2/assignment_loss‚0.49318185889870253 3loss/FBt1/assignment_loss‚0.0003754270595284955 % _timestamp‚1.7843712386144686e+09 2loss/SRqqt1/assignment_loss‚0.6206237947274005 2loss/FBt2/assignment_loss‚0.029806642978628204 _runtime‚3180.058120246 ôÜ Ò… -loss/FRt1/detection_loss‚0.58379699519682 epoch‚8 trainer/global_step‚6749 0loss/FRt1/assignment_loss‚1.6758065719837971 /loss/FRt2/detection_loss‚1.6352610768905531 0loss/SRqqt1/detection_loss‚0.887801548888363 1loss/SRqqt2/detection_loss‚0.7084457473183045 /loss/FBt1/detection_loss‚0.7301587419850486 .loss/FBt2/detection_loss‚0.566276379479942 $loss/total_loss‚8.71680473513026 _step‚547 0loss/FRt2/assignment_loss‚0.7852699497231731 3loss/SRqqt2/assignment_loss‚0.49318185889870253 3loss/FBt1/assignment_loss‚0.0003754270595284955 % _timestamp‚1.7843712386144686e+09 2loss/SRqqt1/assignment_loss‚0.6206237947274005 2loss/FBt2/assignment_loss‚0.029806642978628204 _runtime‚3180.058120246£&¥†Ó‚(:… µ°íұ̪Š gpu.0.gpu‚62.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚116.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.297 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÉY¨ÝÔ‚(:Ò µ°íÒŽŒøŒ* disk./.usagePercent‚39.614676762429596 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 cpu‚26.86650960907209 disk.rbd1.out‚7.8125# disk.nvme0c0n1.in‚ 4166.45703125 disk.rbd0.out‚0" disk.dm-0.out‚5110.08837890625& proc.memory.availableMB‚ 132.703125# proc.memory.rssMB‚ 2692.72265625 network.recv‚ 1.031132e+06 disk.dm-1.out‚ 73.87890625 disk.dm-0.in‚ 4166.45703125 disk.sda.in‚0.109375 disk.rbd0.in‚0$ disk./.usageGB‚590.2873115539551 disk.sda3.in‚0.109375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚5110.08837890625! disk.nvme0n1.in‚ 4166.45703125$ memory_percent‚99.19004440307617) proc.memory.percent‚16.43507480621338 disk.sda3.out‚ 73.87890625 disk.sda.out‚ 73.87890625 network.sent‚ 9.055365e+06% disk.nvme0n1.out‚5110.08837890625¯FÊ2Õ- _wandbruntime‚3201  _runtime‚3201÷³oÅÖ¿ _step‚548 % lr-AdamW/pg1‚0.000800124410964763 % lr-AdamW/pg2‚0.000800124410964763 trainer/global_step‚6799 % _timestamp‚1.7843712602396688e+09 _runtime‚3201.683583729ˆéa¹Ê×Ä % lr-AdamW/pg1‚0.000800124410964763 % lr-AdamW/pg2‚0.000800124410964763 trainer/global_step‚6799 % _timestamp‚1.7843712602396688e+09 _runtime‚3201.683583729 _step‚548¤[ ´2Ø- _wandbruntime‚3201  _runtime‚3201U)ºóÔ'ÙÎO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50982973 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255676710029312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31329285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248871515960832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4008424854016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64507898 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64507898 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4008424854016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259670923003998 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248871308461568 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4008336916992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115500848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50982971 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3994230014046 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4008336916992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46751608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4141576881664 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚3201.684929561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264059435258368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260051115380736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115490869 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260051098341376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46751849 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244729914175488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3994316151136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50982971 % _timestamp‚1.7843712602408266e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31329285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248871515960832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4008424854016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64507898 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64507898 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4008424854016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259670923003998 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248871308461568 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4008336916992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115500848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50982971 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freedÐêfTx‚ 3994230014046 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4008336916992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46751608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4141576881664 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚3201.684929561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264059435258368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260051115380736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115490869 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260051098341376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46751849 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244729914175488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3994316151136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50982971 % _timestamp‚1.7843712602408266e+09 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248906807205376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3994797335576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31333769 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260088209305600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260088209305600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115517364 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115507371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4008994220544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64517179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4141981515264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64517179 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248907010503168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244764825690112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264097220565504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50990194 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50990194 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259708065537418 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46758230 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244764850593792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50990192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115517364 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264097220565504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115507371 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259707962360856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15424696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255713165025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4008906276352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15424461 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4008994220544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192|áñæâ&ÝÜM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64517179 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248907010503168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244764825690112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264097220565504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50990194 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50990194 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259708065537418 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46758230 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244764850593792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50990192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115517364 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264097220565504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115507371 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259707962360856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15424696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255713165025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4008906276352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15424461 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4008994220544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255713182064640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4008906276352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64527170 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64527170 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248906807205376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3994797335576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31333769 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260088209305600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260088209305600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115517364 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115507371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4008994220544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64517179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4141981515264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¦‡©˜2Þ- _wandbruntime‚3202  _runtime‚3202ýžI߀ 0loss/FRt1/assignment_loss‚1.7459439844166467 2loss/SRqqt1/assignment_loss‚0.5876941381838073 2loss/SRqqt2/assignment_loss‚0.6400798879174592 /loss/FRt1/detection_loss‚0.5883410310304222 /loss/FBt2/detection_loss‚0.6769881655852641 %loss/total_loss‚9.183012010861667 _runtime‚3202.030938767 1loss/FBt1/assignment_loss‚0.05936933630814268 1loss/SRqqt1/detection_loss‚0.8826148098175015 1loss/SRqqt2/detection_loss‚0.6652166184942032 .loss/FBt1/detection_loss‚0.887133778275355 2loss/FBt2/assignment_loss‚0.005073123710061161 /loss/FRt2/detection_loss‚1.5921467349738425 epoch‚8 trainer/global_step‚6799 % _timestamp‚1.7843712605873299e+09 /loss/FRt2/assignment_loss‚0.852410402148962 _step‚551|ŽX± à… 0loss/FRt1/assignment_loss‚1.7459439844166467 2loss/SRqqt1/assignment_loss‚0.5876941381838073 2loss/SRqqt2/assignment_loss‚0.6400798879174592 /loss/FRt1/detection_loss‚0.5883410310304222 /loss/FBt2/detection_loss‚0.6769881655852641 %loss/total_loss‚9.183012010861667 _runtime‚3202.030938767 1loss/FBt1/assignment_loss‚0.05936933630814268 1loss/SRqqt1/detection_loss‚0.8826148098175015 1loss/SRqqt2/detection_loss‚0.6652166184942032 .loss/FBt1/detection_loss‚0.887133778275355 2loss/FBt2/assignment_loss‚0.005073123710061161 /loss/FRt2/detection_loss‚1.5921467349738425 epoch‚8 trainer/global_step‚6799 % _timestamp‚1.7843712605873299e+09 /loss/FRt2/assignment_loss‚0.852410402148962 _step‚551§àö(‘á‚(:† Ä°íÒŸ‚΋ gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¯˜\Èâ‚(:½ Ä°íÒíýçŒ disk.sda2.out‚0& proc.memory.availableMB‚ 133.921875 network.recv‚ 1.036001e+06 disk.sda1.out‚0 disk.sda3.out‚74.25 cpu‚26.863809913808126 proc.cpu.threads‚17 disk.rbd1.in‚0$ disk./.usageGB‚590.2884140014648 disk.rbd0.out‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 4166.45703125 disk.drbd2165.in‚0# memory_percent‚99.1826057434082 network.sent‚ 9.107588e+06 disk.dm-1.out‚74.25 disk.sda3.in‚0.109375 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚5114.3681640625 disk.sda2.in‚0! disk.dm-0.out‚5114.3681640625# disk.nvme0c0n1.in‚ 4166.45703125* disk./.usagePercent‚39.614750748606994! disk.nvme0n1.in‚ 4166.45703125 disk.sda1.in‚0 disk.sda.out‚74.25# proc.memory.rssMB‚ 2692.72265625 disk.rbd0.in‚0 disk.sda.in‚0.109375 disk.rbd1.out‚7.8125) proc.memory.percent‚16.43507480621338$ disk.nvme0n1.out‚5114.3681640625ìW÷ß2ã- _wandbruntime‚3223  _runtime‚3223:DÞiÇäÁ & lr-AdamW/pg1‚0.0007993376063918706 & lr-AdamW/pg2‚0.0007993376063918706 trainer/global_step‚6849 % _timestamp‚1.7843712820727417e+09 _runtime‚3223.516557361 _step‚552ôkù;ÌåÆ _step‚552 & lr-AdamW/pg1‚0.0007993376063918706 & lr-AdamW/pg2‚0.0007993376063918706 trainer/global_step‚6849 % _timestamp‚1.7843712820727417e+09 _runtime‚3223.516557361¨pÎ\2æ- _wandbruntime‚3223  _runtime‚3223äã@ð¢ çÎO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250646276892160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51344021 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4036837142016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4170786985472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257500294756352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31553485 _step‚553 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47085802 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250646467614208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843712820738094e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64983777 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116327800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocŸJî¤2ated‚ 4170952803840 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64973800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31553488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15532317 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64973800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚261523025926556 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265943483694592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15532548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚6849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265943483694592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246475514810368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3223.517792022 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51344023 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261906646552576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4036925079040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47086036 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246475489906688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261522922750106 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51344023 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4036925079040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0NÞÙ'èÓO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47085802 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250646467614208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843712820738094e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64983777 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116327800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4170952803840 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64973800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31553488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15532317 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64973800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚261523025926556 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265943483694592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15532548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚6849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265943483694592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246475514810368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3223.517792022 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51344023 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261906646552576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4036925079040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47086036 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246475489906688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261522922750106 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51344023 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4036925079040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250646276892160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51344021 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4036837142016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4170786985472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257500294756352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31553485 _step‚553©•bÑ»2é- _wandbruntime‚3223  _runtime‚3223lV$°Þ&êØM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261943757516800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51351244 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47092645 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265981269001728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116344316 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116334323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15534673 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4037494445568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4037406501376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250681774882304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4171547660800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250681973985792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4171373460992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47092419 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51351244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 _runtime‚3223.863904807 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265981164018176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261943757516800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _step‚554 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51351242 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚261559962106964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246510426324992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257536783831040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64983081 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4037406501376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257536766791680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64993072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261943774556160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15534450 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116344316 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261943774556160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0øe@„ÂëÝM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265981164018176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261943757516800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _step‚554 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51351242 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚261559962106964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246510426324992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257536783831040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64983081 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4037406501376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257536766791680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64993072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261943774556160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15534450 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116344316 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261943774556160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261943757516800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51351244 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47092645 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265981269001728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116344316 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116334323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15534673 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4037494445568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4037406501376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250681774882304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4171547660800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250681973985792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4171373460992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47092419 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51351244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 _runtime‚3223.863904807ª ûzd2ì- _wandbruntime‚3223  _runtime‚3223‹ëÇÓíÿ %loss/total_loss‚8.461149243818564 trainer/global_step‚6849 2loss/SRqqt1/assignment_loss‚0.6532824732965622 /loss/FRt1/detection_loss‚0.5534521254747498 1loss/SRqqt2/detection_loss‚0.6351511303569705 2loss/SRqqt2/assignment_loss‚0.5144544522519925 1loss/FBt1/assignment_loss‚0.07513227841713359 /loss/FBt1/detection_loss‚0.7005391891300679 epoch‚8 % _timestamp‚1.7843712824214578e+09 _runtime‚3223.864990978 /loss/FRt1/assignment_loss‚1.723136278932745 /loss/FRt2/assignment_loss‚0.641816841058082 1loss/FBt2/assignment_loss‚0.06179724626346835 1loss/SRqqt1/detection_loss‚0.8468671001923772 _step‚555 /loss/FRt2/detection_loss‚1.5080079297042641 /loss/FBt2/detection_loss‚0.5475121987401508L/r î„ 2loss/SRqqt2/assignment_loss‚0.5144544522519925 1loss/FBt1/assignment_loss‚0.07513227841713359 /loss/FBt1/detection_loss‚0.7005391891300679 epoch‚8 % _timestamp‚1.7843712824214578e+09 _runtime‚3223.864990978 /loss/FRt1/assignment_loss‚1.723136278932745 /loss/FRt2/assignment_loss‚0.641816841058082 1loss/FBt2/assignment_loss‚0.06179724626346835 1loss/SRqqt1/detection_loss‚0.8468671001923772 _step‚555 /loss/FRt2/detection_loss‚1.5080079297042641 /loss/FBt2/detection_loss‚0.5475121987401508 %loss/total_loss‚8.461149243818564 trainer/global_step‚6849 2loss/SRqqt1/assignment_loss‚0.6532824732965622 /loss/FRt1/detection_loss‚0.5534521254747498 1loss/SRqqt2/detection_loss‚0.6351511303569705«£1ð’‘ï‚(:† Ó°íÒ³™âŠ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚137.511' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØiPØð‚(:Í Ó°íÒ©‚Ó! disk.nvme0n1.in‚ 4252.06640625$ disk.nvme0n1.out‚5120.5322265625( proc.memory.availableMB‚ 133.87109375 network.sent‚ 9.183012e+06 disk.sda2.in‚0 disk.drbd2165.in‚0$ memory_percent‚99.18291568756104 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2692.72265625) proc.memory.percent‚16.43507480621338 disk.sda2.out‚0 disk.dm-1.out‚ 74.4921875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 4252.06640625& disk.nvme0c0n1.out‚5120.5322265625 disk.sda.in‚0.109375* disk./.usagePercent‚39.614747164501516$ disk./.usageGB‚590.2883605957031 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 4252.06640625 cpu‚26.86088825069149 network.recv‚ 1.041121e+06! disk.dm-0.out‚5120.5322265625 disk.sda3.in‚0.109375 disk.sda.out‚ 74.4921875 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚7.875 disk.rbd0.out‚0 disk.sda3.out‚ 74.4921875Eœ:´œñ‚(:‘ â°íÒµãÈŠ gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…™!Ýò‚(:Ò â°íÒçÜéŒ( proc.memory.availableMB‚ 133.87109375 disk.sda1.in‚0 disk.dm-1.out‚ 74.76953125 disk.sda3.in‚0.109375 disk.sda3.out‚ 74.76953125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚5123.81298828125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.in‚0.109375$ memory_percent‚99.18291568756104# proc.memory.rssMB‚ 2692.72265625 network.sent‚ 9.232891e+06 disk.sda2.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4252.06640625) proc.memory.percent‚16.43507480621338 disk.dm-1.in‚0.109375" disk.dm-0.out‚5123.81298828125 disk.sda.out‚ 74.76953125 proc.cpu.threads‚17 network.recv‚ 1.046572e+06) disk./.usagePercent‚39.61474050830562! disk.nvme0n1.in‚ 4252.06640625$ disk./.usageGB‚590.2882614135742 disk.rbd0.out‚0 disk.dm-0.in‚ 4252.06640625 disk.rbd1.out‚7.875% disk.nvme0n1.out‚5123.81298828125 cpu‚26.858021344197931oŠ2ó- _wandbruntime‚3245  _runtime‚32459ƒVWÇôÁ trainer/global_step‚6899 % _timestamp‚1.7843713041894088e+09 _runtime‚3245.633213944 _step‚556 & lr-AdamW/pg1‚0.0007985447905116331 & lr-AdamW/pg2‚0.0007985447905116331×aù\ÌõÆ & lr-AdamW/pg1‚0.0007985447905116331 & lr-AdamW/pg2‚0.0007985447905116331 trainer/global_step‚6899 % _timestamp‚1.7843713041894088e+09 _runtime‚3245.633213944 _step‚556¬­ˆÄU2ö- _wandbruntime‚3245  _runtime‚3245·k7úÒ'÷ÌO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51705071 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263375025672664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4065425304064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4200223204352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4051025973462 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚557 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4065425304064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65439702 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252421485855744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267827637107200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4065337367040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263762211803136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252421288842240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4051112110552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47419896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15641996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267827532130816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267827532130816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263762194763776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51705073 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47419681 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248221090541568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263374922496214 trainer/global_step‚6899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263762211803136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 3245.63455199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65449679 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4200395314176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263762194763776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4065337367040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65439702 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15642208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31777688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259323913562112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 Cš×'øÑO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51705073 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51705071 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117154752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65449679 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31777685 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263762211803136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252421288842240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4051112110552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47419896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15641996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267827532130816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267827532130816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263762194763776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51705073 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47419681 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248221090541568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263374922496214 trainer/global_step‚6899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263762211803136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 3245.63455199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65449679 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4200395314176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263762194763776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4065337367040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65439702 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15642208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31777688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259323913562112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51705071 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263375025672664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4065425304064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4200223204352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4051025973462 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚557 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4065425304064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65439702 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252421485855744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267827637107200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4065337367040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267865212454400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252456782003200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4065906726400 trainer/global_step‚6899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51712292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263799322767360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263412065029634 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117161275 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15644401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252456991592448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259360368558080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263799305728000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065994670592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4065994670592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31782169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47426346 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4051679432194 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4051593294992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267865317437952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51712294 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65448983 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4065906726400 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 % _timestamp‚1.7843713045329015e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200989536256 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267865317437952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248256002056192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65458974 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117171268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31782172 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117171268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117161275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15644177 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263411961853072 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65458974 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51712294 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259360385597440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936€«­²ã&ûÝM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200989536256 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267865317437952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248256002056192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65458974 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117171268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31782172 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117171268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117161275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15644177 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263411961853072 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65458974 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51712294 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259360385597440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263799322767360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263799305728000 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚3245.976955712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267865212454400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252456782003200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4065906726400 trainer/global_step‚6899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51712292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263799322767360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263412065029634 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117161275 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15644401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252456991592448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259360368558080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263799305728000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065994670592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4065994670592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31782169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47426346 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4051679432194 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4051593294992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267865317437952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51712294 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65448983 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4065906726400 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 % _timestamp‚1.7843713045329015e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552®Ô½â12ü- _wandbruntime‚3245  _runtime‚3245Ñž§öýÿ $ _timestamp‚1.784371304534883e+09 0loss/SRqqt2/detection_loss‚0.704314195693904 /loss/FBt1/detection_loss‚0.8090487003326416 /loss/FBt2/detection_loss‚0.5039281497334337 epoch‚8 _step‚559 1loss/SRqqt1/assignment_loss‚0.640268013573512 1loss/FBt2/assignment_loss‚0.07420055170797389 0loss/SRqqt1/detection_loss‚0.777902560553924 trainer/global_step‚6899 0loss/FRt1/assignment_loss‚1.7416645169258118 2loss/SRqqt2/assignment_loss‚0.5501210948280612 3loss/FBt1/assignment_loss‚0.0055199990692472055 %loss/total_loss‚8.792571188800222 _runtime‚3245.978428493 0loss/FRt2/assignment_loss‚0.7741808685216498 /loss/FRt1/detection_loss‚0.5660984168288661 /loss/FRt2/detection_loss‚1.6453241210311969[mí‘ þ„ 0loss/FRt1/assignment_loss‚1.7416645169258118 2loss/SRqqt2/assignment_loss‚0.5501210948280612 3loss/FBt1/assignment_loss‚0.0055199990692472055 %loss/total_loss‚8.792571188800222 _runtime‚3245.978428493 0loss/FRt2/assignment_loss‚0.7741808685216498 /loss/FRt1/detection_loss‚0.5660984168288661 /loss/FRt2/detection_loss‚1.6453241210311969 $ _timestamp‚1.784371304534883e+09 0loss/SRqqt2/detection_loss‚0.704314195693904 /loss/FBt1/detection_loss‚0.8090487003326416 /loss/FBt2/detection_loss‚0.5039281497334337 epoch‚8 _step‚559 1loss/SRqqt1/assignment_loss‚0.640268013573512 1loss/FBt2/assignment_loss‚0.07420055170797389 0loss/SRqqt1/detection_loss‚0.777902560553924 trainer/global_step‚6899¯ylÿ‚(:… ñ°íÒÄŒýŠ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚193.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.6262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ÇpÚ€‚(:Ï ñ°íÒ¾·ƒ disk.sda2.in‚0) proc.memory.percent‚16.43507480621338 proc.cpu.threads‚17$ disk./.usageGB‚590.2883605957031 disk.sda.in‚0.109375 disk.sda1.in‚0 network.sent‚ 9.237931e+06* disk./.usagePercent‚39.614747164501516 disk.drbd2165.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2692.72265625 cpu‚26.85508733644188 network.recv‚ 1.050462e+06" disk.dm-0.out‚5129.48583984375 disk.sda.out‚ 75.07421875' disk.nvme0c0n1.out‚5129.48583984375# proc.memory.availableMB‚133.875 disk.sda3.in‚0.109375 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 4252.06640625 disk.rbd0.out‚0 disk.dm-1.out‚ 75.07421875$ memory_percent‚99.18289184570312 disk.dm-0.in‚ 4252.06640625 disk.sda3.out‚ 75.07421875! disk.nvme0n1.in‚ 4252.06640625% disk.nvme0n1.out‚5129.48583984375 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚7.9375$Èðý2- _wandbruntime‚3267  _runtime‚3267_È0àÄ‚¾ % lr-AdamW/pg1‚0.000797745976022606 % lr-AdamW/pg2‚0.000797745976022606 trainer/global_step‚6949 $ _timestamp‚1.784371326079846e+09 _runtime‚3267.523671675 _step‚560ø3«ðɃà _step‚560 % lr-AdamW/pg1‚0.000797745976022606 % lr-AdamW/pg2‚0.000797745976022606 trainer/global_step‚6949 $ _timestamp‚1.784371326079846e+09 _runtime‚3267.523671675°¸Q["2„- _wandbruntime‚3267  _runtime‚3267¿°´«Ô'…ÎO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4093837592064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65905604 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52066121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47753753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261147515328512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265617760014336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265617760014336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254196554035712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52066123 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265617742974976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4079423953170 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269711580567040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47753535 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265226922242322 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65905604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269711580567040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4093925529088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15751865 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254196363313664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4093925529088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269711685543424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32001885 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _step‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚218 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4229721944576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254196554035712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52066123 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265617742974976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4079423953170 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269711580567040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47753535 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265226922242322 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZLDeviceStatsMonitor.Ç`a\ëon_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65905604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269711580567040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4093925529088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15751865 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254196363313664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4093925529088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269711685543424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32001885 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _step‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚218 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4229721944576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265654853939200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117988227 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47760179 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250001552883712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52073344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52073342 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261183987363840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117998220 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4094406951424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4094406951424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265263961599180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65924876 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265654870978560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843713264199116e+09 _step‚562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65924876 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117998220 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265654853939200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269749365874176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4230480379392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ž8Ýã&‰ÝM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4094494895616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52073342 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230308276736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269749260890624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4079991274700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265654870978560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47760407 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 _runtime‚3267.863930641 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4094494895616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117988227 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4080077411902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15753810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32006372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32006369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65914885 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52073344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65914885 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265654853939200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117988227 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47760179 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250001552883712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52073344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52073342 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261183987363840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117998220 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4094406951424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4094406951424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265263961599180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65924876 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265654870978560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843713264199116e+09 _step‚562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65924876 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117998220 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265654853939200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269749365874176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4230480379392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271595629003264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52427171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚565 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15862194 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267473308225536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271595733979648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚565 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15862194 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267473308225536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271595733979648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271595629003264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52427171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271633414310400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52434394 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271633309326848 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _step‚566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251747128614912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267510402150400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32230569 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66380787 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251747153518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256006888804352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4122995120640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15864410 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271633414310400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118825172 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4122907176448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48094979 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4108475391610 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32230572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267510419189760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52434392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15864641 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4259938583552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267116064521850 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263007589130240e»ßý½™ÜM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267510419189760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52434392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15864641 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4259938583552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267116064521850 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263007589130240 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118825172 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52434394 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263007572090880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256007092102144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚3289.875595749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118815179 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚ Q\%0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271633414310400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52434394 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271633309326848 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 _step‚566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251747128614912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267510402150400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32230569 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66380787 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251747153518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256006888804352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4122995120640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15864410 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271633414310400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118825172 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4122907176448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48094979 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4108475391610 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32230572¶mI==2š- _wandbruntime‚3289  _runtime‚3289È™ª ›ƒ 0loss/FRt1/assignment_loss‚1.6535481362976832 2loss/SRqqt1/assignment_loss‚0.6193576382711098 2loss/FBt1/assignment_loss‚0.014210078529085226 /loss/FRt2/detection_loss‚1.4692326890295508 /loss/FBt2/detection_loss‚0.7796478768226357 0loss/FRt2/assignment_loss‚0.7187482374055045 2loss/SRqqt2/assignment_loss‚0.5028297893395547 1loss/SRqqt1/detection_loss‚0.8770326138474047 epoch‚8 3loss/FBt2/assignment_loss‚0.0004104800009206171 0loss/SRqqt2/detection_loss‚0.585064646620707 %loss/total_loss‚8.347752351442402 trainer/global_step‚6999 _step‚567 /loss/FRt1/detection_loss‚0.5236204958769684 /loss/FBt1/detection_loss‚0.6040496694012767 % _timestamp‚1.7843713484331746e+09 _runtime‚3289.876712848Äe月 0loss/FRt1/assignment_loss‚1.6535481362976832 2loss/SRqqt1/assignment_loss‚0.6193576382711098 2loss/FBt1/assignment_loss‚0.014210078529085226 /loss/FRt2/detection_loss‚1.4692326890295508 /loss/FBt2/detection_loss‚0.7796478768226357 0loss/FRt2/assignment_loss‚0.7187482374055045 2loss/SRqqt2/assignment_loss‚0.5028297893395547 1loss/SRqqt1/detection_loss‚0.8770326138474047 epoch‚8 3loss/FBt2/assignment_loss‚0.0004104800009206171 0loss/SRqqt2/detection_loss‚0.585064646620707 %loss/total_loss‚8.347752351442402 trainer/global_step‚6999 _step‚567 /loss/FRt1/detection_loss‚0.5236204958769684 /loss/FBt1/detection_loss‚0.6040496694012767 % _timestamp‚1.7843713484331746e+09 _runtime‚3289.876712848·O` –œ‚(:‘ ž±íÒìÁƒŒ gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚189.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öó^Ôž‚(:É ž±íÒÛϼ disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚5140.66455078125# proc.memory.rssMB‚ 2692.72265625 disk.sda1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 133.54296875' disk.nvme0c0n1.out‚5140.66455078125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 4345.8984375% disk.nvme0n1.out‚5140.66455078125 disk.rbd0.in‚0 disk.dm-0.in‚ 4345.8984375 disk.sda3.in‚0.109375) proc.memory.percent‚16.43507480621338 cpu‚26.846972388481987 network.recv‚ 1.064236e+06$ disk./.usageGB‚590.2890243530273 disk.nvme0n1.in‚ 4345.8984375 disk.sda.out‚75.8125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚75.8125 network.sent‚ 9.368388e+06) disk./.usagePercent‚39.61479170981247 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.out‚ 8.00390625$ memory_percent‚99.18491840362549 disk.dm-1.in‚0.109375 disk.dm-1.out‚75.8125'±– 2Ÿ- _wandbruntime‚3311  _runtime‚3311Çx6Á » # lr-AdamW/pg1‚0.0007961304024926 # lr-AdamW/pg2‚0.0007961304024926 trainer/global_step‚7049 % _timestamp‚1.7843713699037154e+09 _runtime‚3311.347612021 _step‚568‘ ë·Æ¡À _step‚568 # lr-AdamW/pg1‚0.0007961304024926 # lr-AdamW/pg2‚0.0007961304024926 trainer/global_step‚7049 % _timestamp‚1.7843713699037154e+09 _runtime‚3311.347612021¸c'k2¢- _wandbruntime‚3311  _runtime‚3311„ìNÔ'£ÎO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32450285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257746662863360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273479677439488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52788223 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264794701821952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4150925979136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66847385 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15972314 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3311.348899541 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264794718861312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269328839397376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32450288 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273479677439488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52788223 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264794701821952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4150925979136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66847385 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979¹˜æ2¥- _wandbruntime‚3311  _runtime‚33110Åcö¦×M EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119652124 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268968064267958 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4136787234116 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ;J$ÈÊç"5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119642131 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66856680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273517357763072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269365950361600 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119652124 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66846689 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66856680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32454772 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4151495345664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119642131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 trainer/global_step‚7049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32454769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15974700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784371370246789e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52795444 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273517462746624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253492704346112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66846689 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269365967400960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48429236 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4289257449984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264831190896640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4136873371318 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273517462746624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257782162996736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257781961796096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4151495345664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268967961091396 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3311.690940405 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0³[¯Œâ&§ÜM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _step‚570 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32454772 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4151495345664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119642131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 trainer/global_step‚7049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32454769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15974700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784371370246789e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52795444 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273517462746624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253492704346112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66846689 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269365967400960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48429236 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4289257449984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264831190896640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4136873371318 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273517462746624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257782162996736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257781961796096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4151495345664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268967961091396 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3311.690940405 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119652124 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268968064267958 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4136787234116 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119642131 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66856680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273517357763072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269365950361600 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119652124 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66846689 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66856680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32674485 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32674488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48756049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚266618320627712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48755816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4318156131840 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271184404647936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271184387608576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67303310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4179338267136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16081331 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚7099 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271184404647936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67313287 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚{Äl 473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53149271 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120452581 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53149273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255203393466368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4164617892294 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275363725875712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120462560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚266618303588352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67303310 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0üÖÒÙ'´ÓO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4179426204160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255203368562688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271184387608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32674485 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32674488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48756049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚266618320627712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48755816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4318156131840 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271184404647936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271184387608576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67303310 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4179338267136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16081331 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚7099 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271184404647936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67313287 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53149271 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120452581 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53149273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255203393466368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4164617892294 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275363725875712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120462560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚266618303588352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67303310 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275363725875712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53149273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271221515612160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4179907626496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚3333.731173733 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120469083 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67312591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4179995570688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120469083 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4318747006464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48762515 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67322582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271221498572800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120479076 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255238280077312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4165185213824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4179907626496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4318919109120 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53156492 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53156492 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259557224090112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275401511182848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16083546 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275401511182848 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120479076 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154¤»Äã&·ÝM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16083546 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275401511182848 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120479076 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275401406199296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270820064014066 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67322582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270819960837504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67312591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255238304980992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53156494 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271221515612160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48762758 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271221515612160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4179907626496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚3333.731173733 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120469083 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67312591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4179995570688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120469083 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4318747006464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48762515 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67322582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271221498572800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120479076 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255238280077312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4165185213824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4179907626496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4318919109120 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53156492 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53156492 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259557224090112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275401511182848¾8ºÀ2¸- _wandbruntime‚3333  _runtime‚3333‡y³R¹‚ 1loss/SRqqt1/detection_loss‚0.8562286208369263 %loss/total_loss‚8.272482283657064 epoch‚8 1loss/SRqqt2/detection_loss‚0.6356455723850438 trainer/global_step‚7099 % _timestamp‚1.7843713922887204e+09 /loss/FRt2/assignment_loss‚0.665984682218973 0loss/FBt1/detection_loss‚0.36908650977743995 _runtime‚3333.732249554 _step‚575 0loss/FRt1/assignment_loss‚1.6836935139464257 2loss/SRqqt1/assignment_loss‚0.7364687720763646 2loss/FBt1/assignment_loss‚0.041524477186443046 1loss/FBt2/assignment_loss‚0.06254016568880172 .loss/FRt1/detection_loss‚0.591259610962346 /loss/FRt2/detection_loss‚1.5857194086027786 /loss/FBt2/detection_loss‚0.5654217252006325 3loss/SRqqt2/assignment_loss‚0.47890922477488923ñÅ–µº‡ 3loss/SRqqt2/assignment_loss‚0.47890922477488923 1loss/SRqqt1/detection_loss‚0.8562286208369263 %loss/total_loss‚8.272482283657064 epoch‚8 1loss/SRqqt2/detection_loss‚0.6356455723850438 trainer/global_step‚7099 % _timestamp‚1.7843713922887204e+09 /loss/FRt2/assignment_loss‚0.665984682218973 0loss/FBt1/detection_loss‚0.36908650977743995 _runtime‚3333.732249554 _step‚575 0loss/FRt¾”åX1/assignment_loss‚1.6836935139464257 2loss/SRqqt1/assignment_loss‚0.7364687720763646 2loss/FBt1/assignment_loss‚0.041524477186443046 1loss/FBt2/assignment_loss‚0.06254016568880172 .loss/FRt1/detection_loss‚0.591259610962346 /loss/FRt2/detection_loss‚1.5857194086027786 /loss/FBt2/detection_loss‚0.5654217252006325¿9&}¶‘»‚(:† ˱íÒ­¢™Š gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚187.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐËÜ£Ô¼‚(:É ˱íÒéïÐŒ network.recv‚ 1.077091e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 4469.79296875 disk.sda.out‚76.59375 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚76.59375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 4469.79296875 cpu‚26.838799104581025 proc.cpu.threads‚17 disk.sda3.in‚0.109375( proc.memory.availableMB‚ 133.79296875# proc.memory.rssMB‚ 2692.72265625) proc.memory.percent‚16.43507480621338$ disk./.usageGB‚590.2892036437988 disk.rbd1.out‚ 8.20703125% disk.nvme0c0n1.out‚5153.224609375$ memory_percent‚99.18339252471924 disk.sda3.out‚76.59375) disk./.usagePercent‚39.61480374216658 disk.dm-0.out‚5153.224609375 network.sent‚ 9.497571e+06! disk.nvme0n1.in‚ 4469.79296875# disk.nvme0n1.out‚5153.224609375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0.109375¹L2½- _wandbruntime‚3355  _runtime‚3355Te]uǾÁ & lr-AdamW/pg1‚0.0007944909893081952 & lr-AdamW/pg2‚0.0007944909893081952 trainer/global_step‚7149 % _timestamp‚1.7843714139627433e+09 _runtime‚3355.406602844 _step‚576œœ…®Ì¿Æ _runtime‚3355.406602844 _step‚576 & lr-AdamW/pg1‚0.0007944909893081952 & lr-AdamW/pg2‚0.0007944909893081952 trainer/global_step‚7149 % _timestamp‚1.7843714139627433e+09ÀwTÝ2À- _wandbruntime‚3355  _runtime‚3355™æ™ÛÔ'ÁÎO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277247879288320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67779189 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4207838492160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277247774311936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53510323 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53510321 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4207926429184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273039935819776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16190830 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121279533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4347651286528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272635024403204 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4193015872002 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49089515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49089743 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚3355.407883313 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53510323 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67779189 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚7149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67769212 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272634921226754 _step‚577 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256948944293888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268441905354752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268441922394112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32898688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚273039952859136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256948969197568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843714139638796e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4207838492160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277247774311936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67769212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚273039935819776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16191055 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚7149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67769212 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272634921226754 _step‚577 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256948944293888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268441905354752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268441922394112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32898688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚273039952859136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256948969197568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843714139638796e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4207838492160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277247774311936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67769212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚273039935819776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16191055 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277247774311936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53510323 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53510321 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4207926429184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273039935819776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16190830 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121279533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4347651286528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272635024403204 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4193015872002 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679Á#DU2Ã- _wandbruntime‚3355  _runtime‚3355–Ãí†Þ&ÄØM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53517544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272672063760174 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268478394429440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67788484 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚3355.749174109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273077063823360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16193264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121296035 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67778493 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4193583193532 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4208495795712 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚7149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4208407851520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 % _timestamp‚1.7843714143051443e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32903169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256983880712192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261332106534400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67788484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273077063823360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚261332288860672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268478377390080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273077046784000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67778493 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121296035 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4208495795712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182326272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121306028 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121306028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4208407851520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4348250725888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49096436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32903172 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4348408148480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53517542 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53517542 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53517544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49096206 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277285559619072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277285559619072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277285454635520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16193037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4193669330734 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚273077046784000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154UŽ‚;ÅÝM KEDeviceStatsMonitor.on_train_batch_end/reserved_byš½b,¨&tes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49096436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32903172 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4348408148480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53517542 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53517542 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53517544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49096206 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277285559619072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277285559619072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277285454635520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16193037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4193669330734 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚273077046784000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53517544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272672063760174 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268478394429440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67788484 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚3355.749174109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273077063823360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16193264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121296035 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67778493 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4193583193532 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4208495795712 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚7149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4208407851520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 % _timestamp‚1.7843714143051443e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32903169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256983880712192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261332106534400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67788484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273077063823360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚261332288860672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268478377390080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273077046784000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67778493 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121296035 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4208495795712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182326272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121306028 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121306028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4208407851520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4348250725888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4377345280000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4236338717184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263071890208768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274486920972862 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122116464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7199 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16301518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258694544928768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4236426654208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274487024149312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270265507121152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4221413851710 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚3377.146168461 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274895501070336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33122885 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274895484030976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49424403 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33122888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚258694520025088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279131822748160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270265524160512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53871373 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4236426654208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68245091 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279131927724544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122106485 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279131822748160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53871371 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4377168975872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53871371 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0››UýÒÓO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4221499988800 _step‚581 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279131822748160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53871371 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4377168975872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53871371 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68245091 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274895501070336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68235114 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49424635 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263071689000960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843714357021973e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16301747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279131927724544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68235114 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4377345280000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4236338717184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263071890208768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274486920972862 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122116464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7199 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16301518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258694544928768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4236426654208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274487024149312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270265507121152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4221413851710 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚3377.146168461 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274895501070336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33122885 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274895484030976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49424403 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33122888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚258694520025088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279131822748160 KEDeviceStatsMonitor.on_train_batch_start/allocatiêÓ Ü on.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270265524160512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53871373 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4236426654208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68245091 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279131927724544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122106485 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270301996195840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68254386 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122122987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263107389551616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33127369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68244395 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279169608055296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68244395 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274932612034560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4221981173240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270301979156480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚ 3377.48512228 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53878592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4377758908416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4222067310442 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53878592 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49431270 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274524063506282 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274932612034560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274932594995200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274523960329720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53878594 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4236908076544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4236908076544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53878594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 $ _timestamp‚1.784371436041016e+09 _step‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122132980 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122122987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122132980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258729456443392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279169608055296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16303898 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4377933108224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279169503071744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49431033 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49431033 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270301996195840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68254386 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122122987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263107389551616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33127369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68244395 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279169608055296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68244395 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274932612034560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4221981173240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270301979156480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚ 3377.48512228 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53878592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4377758908416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4222067310442 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53878592 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49431270 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274524063506282 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274932612034560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274932594995200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274523960329720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53878594 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4236908076544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4236908076544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53878594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 $ _timestamp‚1.784371436041016e+09 _step‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122132980 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122122987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122132980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258729456443392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279169608055296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16303898 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4377933108224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ƽbà2Ö- _wandbruntime‚3377  _runtime‚3377B r× 1loss/SRqqt1/assignment_loss‚0.640598050841148 1loss/FBt1/assignment_loss‚0.03405628201941605 .loss/FRt2/detection_loss‚1.524218009174641 %loss/total_loss‚8.367851886446516 1loss/SRqqt1/detection_loss‚0.7652974014625095 /loss/FBt2/detection_loss‚0.5092550611472688 epoch‚8 _step‚583 0loss/FRt1/assignment_loss‚1.7666363067953306 0loss/FRt2/assignment_loss‚0.6389878674093725 2loss/FBt2/assignment_loss‚0.046092394841867446 trainer/global_step‚7199 % _timestamp‚1.7843714360427296e+09 _runtime‚3377.486267813 3loss/SRqqt2/assignment_loss‚0.47250367510155933 /loss/FRt1/detection_loss‚0.5824148943029562 1loss/SRqqt2/detection_loss‚0.6328729188556593 /loss/FBt1/detection_loss‚0.7549190244947871æáí ؆ 2loss/FBt2/assignment_loss‚0.046092394841867446 trainer/global_step‚7199 % _timestamp‚1.7843714360427296e+09 _runtime‚3377.486267813 3loss/SRqqt2/assignment_loss‚0.47250367510155933 /loss/FRt1/detection_loss‚0.5824148943029562 1loss/SRqqt2/detection_loss‚0.6328729188556593 /loss/FBt1/detection_loss‚0.7549190244947871 1loss/SRqqt1/assignment_loss‚0.640598050841148 1loss/FBt1/assignment_loss‚0.03405628201941605 .loss/FRt2/detection_loss‚1.524218009174641 %loss/total_loss‚8.367851886446516 1loss/SRqqt1/detection_loss‚0.7652974014625095 /loss/FBt2/detection_loss‚0.5092550611472688 epoch‚8 _step‚583 0loss/FRt1/assignment_loss‚1.7666363067953306 0loss/FRt2/assignment_loss‚0.6389878674093725Ç¡ª@œÙ‚(:‘ ø±íÒºµåŒ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚191.777' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.355399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØLàÚ‚(:Õ ø±íÒ‘±ó disk.drbd2165.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2693.98046875 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 4471.4921875) proc.memory.percent‚16.44275188446045 cpu‚26.84092460530594 disk.dm-0.in‚ 4471.4921875 disk.sda3.in‚0.109375 disk.sda2.in‚0( proc.memory.availableMB‚ 132.64453125 proc.cpu.threads‚17* disk./.usagePercent‚39.614778141413154 disk.dm-1.in‚0.109375 disk.sda3.out‚ 77.47265625 disk.rbd1.out‚ 8.26953125 disk.rbd0.out‚0 disk.sda2.out‚0 network.sent‚ 9.627781e+06 network.recv‚ 1.090862e+06 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚590.2888221740723" disk.dm-0.out‚5166.92431640625% disk.nvme0n1.out‚5166.92431640625" disk.nvme0c0n1.in‚ 4471.4921875' disk.nvme0c0n1.out‚5166.92431640625 disk.dm-1.out‚ 77.47265625$ memory_percent‚99.19040203094482 disk.sda.out‚ 77.47265625Uí>§2Û- _wandbruntime‚3400  _runtime‚3400ýåŠØõÜô{ 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1269226372241974 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34357208013534546 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15962322056293488 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6468220949172974 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6700159311294556 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9033275842666626 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9950953722000122 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6745230555534363 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6574244499206543 +jet/accuracy_3_of_4‚0.08696026355028152 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.147894024848938 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962725043296814 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006708162371069193 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6958474516868591 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7206354737281799 particle/accuracy_2_of_2‚1 )jet/accuracy_1_of_3‚0.891330361366272 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15064403414726257 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991559624671936 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306105494499207 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08905751258134842 +jet/accuracy_2_of_2‚0.14025096595287323 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32543694972991943 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3888888657093048 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6508292555809021 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11538085341453552 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 $ _timestamp‚1.784371459074583e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6261124610900879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17877434194087982 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.37229955196380615 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26258012652397156 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47698819637298584 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5998854637145996 'particle/f_score‚0.6377160549163818 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6198957562446594 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6365917921066284 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0724545568227768 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9926446080207825 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.362026184797287 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03287549316883087 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14292000234127045 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987118124961853 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5906053781509399 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6505171656608582 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30097058415412903 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4739748239517212 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5901014804840088 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10086467117071152 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_1_of_4‚0.9553080797195435 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 +particle/specificity‚0.9048203229904175 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7581058144569397 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08466701954603195 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8346491456031799 _runtime‚3400.519019995 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6247323155403137 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1408075988292694 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017493966966867447 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49000000953674316 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4104938209056854 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9943783283233643 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6121182441711426 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14193931221961975 +jet/accuracy_3_of_3‚0.07537046074867249 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 jet/accuracy_4_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998918890953064 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5927233695983887 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7003050446510315 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8885015845298767 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974288940429688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5906053781509399 >&Purity/1FRt*SRqqt1FBt/event_proportion¸lƒ"‚0.01643880270421505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02077854983508587 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.229823616865094 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.031618427485227585 +jet/accuracy_2_of_3‚0.49813148379325867 (particle/accuracy‚0.8460326194763184 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14763109385967255 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6557492017745972 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0724545568227768 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9910256862640381 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.84906005859375 epoch‚8 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12740521132946014 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996954619884491 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6254397630691528 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5714681148529053 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5930771231651306 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7653845548629761 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08361397683620453 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4997875988483429 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5758787989616394 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1327706128358841 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9960318207740784 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.56827312707901 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 /particle/accuracy_4_of_4‚0.9489232897758484 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16382041573524475 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46590909361839294 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12594330310821533 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11426921933889389 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8346491456031799 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_3_of_5‚1 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986345112323761 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16504451632499695 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21987806260585785 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1275912970304489 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9514853954315186 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11084970086812973 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01178670022636652 particle/accuracy_1_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20755355060100555 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13911548256874084 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6302192807197571 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9974135160446167 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4551209509372711 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.553906261920929 +particle/sensitivity‚0.6310939192771912 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6175869107246399 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6495343446731567 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7533583641052246 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7254571318626404 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 jet/accuracy_1_of_5‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.675216019153595 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1327706128358841 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17291666567325592 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.813247799873352 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7630356550216675 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚584 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.81553715467453 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 trainer/global_step‚7208 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6530773639678955 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13469550013542175 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16674937307834625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08713900297880173 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7630356550216675 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34949907660484314 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6016208529472351 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_2‚0.7242249846458435 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6450657844543457 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959110021591187 ,jet/accuracy_4_of_4‚0.016514042392373085 .particle/accuracy_3_of_3‚0.999514102935791 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13258327543735504 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5980299711227417 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.654097855091095 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7244849801063538 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6423611044883728 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8755430579185486 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 *jet/accuracy_1_of_1‚0.2788497805595398 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.591991662979126 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15490131080150604¡P˜íÿ=Ýù{ ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017493966966867447 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49000000953674316 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4104938209056854 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9943783283233643 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6121182441711426 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14193931221961975 +jet/accuracy_3_of_3‚0.07537046074867249 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 jet/accuracy_4_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998918890953064 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5927233695983887 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7003050446510315 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8885015845298767 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974288940429688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5906053781509399 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02077854983508587 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.229823616865094 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.031618427485227585 +jet/accuracy_2_of_3‚0.49813148379325867 (particle/accuracy‚0.8460326194763184 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14763109385967255 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6557492017745972 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0724545568227768 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9910256862640381 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.84906005859375 epoch‚8 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12740521132946014 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996954619884491 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6254397630691528 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5714681148529053 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5930771231651306 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7653845548629761 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08361397683620453 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4997875988483429 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5758787989616394 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1327706128358841 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9960318207740784 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.56827312707901 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 /particle/accuracy_4_of_4‚0.9489232897758484 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16382041573524475 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46590909361839294 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12594330310821533 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11426921933889389 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8346491456031799 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_3_of_5‚1 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986345112323761 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16504451632499695 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21987806260585785 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1275912970304489 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9514853954315186 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11084970086812973 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01178670022636652 particle/accuracy_1_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20755355060100555 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13911548256874084 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6302192807197571 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9974135160446167 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4551209509372711 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.553906261920929 +particle/sensitivity‚0.6310939192771912 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6175869107246399 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6495343446731567 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7533583641052246 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7254571318626404 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 jet/accuracy_1_of_5‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.675216019153595 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1327706128358841 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17291666567325592 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.813247799873352 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7630356550216675 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _step‚584 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.81553715467453 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 trainer/global_step‚7208 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6530773639678955 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13469550013542175 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16674937307834625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08713900297880173 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7630356550216675 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34949907660484314 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6016208529472351 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_2‚0.7242249846458435 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6450657844543457 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959110021591187 ,jet/accuracy_4_of_4‚0.016514042392373085 .particle/accuracy_3_of_3‚0.999514102935791 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13258327543735504 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5980299711227417 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.654097855091095 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7244849801063538 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6423611044883728 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8755430579185486 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 *jet/accuracy_1_of_1‚0.2788497805595398 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.591991662979126 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15490131080150604 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1269226372241974 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34357208013534546 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15962322056293488 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6468220949172974 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6700159311294556 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9033275842666626 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9950953722000122 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6745230555534363 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6574244499206543 +jet/accuracy_3_of_4‚0.08696026355028152 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.147894024848938 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962725043296814 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006708162371069193 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6958474516868591 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7206354737281799 particle/accuracy_2_of_2‚1 )jet/accuracy_1_of_3‚0.891330361366272 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15064403414726257 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991559624671936 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306105494499207 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08905751258134842 +jet/accuracy_2_of_2‚0.14025096595287323 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32543694972991943 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3888888657093048 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6508292555809021 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11538085341453552 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 $ _timestamp‚1.784371459074583e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6261124610900879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17877434194087982 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.37229955196380615 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26258012652397156 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47698819637298584 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5998854637145996 'particle/f_score‚0.6377160549163818 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6198957562446594 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6365917921066284 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0724545568227768 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9926446080207825 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.362026184797287 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03287549316883087 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14292000234127045 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987118124961853 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5906053781509399 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6505171656608582 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30097058415412903 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4739748239517212 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5901014804840088 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10086467117071152 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_1_of_4‚0.9553080797195435 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 +particle/specificity‚0.9048203229904175 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7581058144569397 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08466701954603195 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8346491456031799 _runtime‚3400.519019995 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6247323155403137 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1408075988292694È×ÕˆÞ‚(:„ ‡²íÒéþØŠ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚177.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@hÛÁß‚(:¶ ‡²íÒñØõŒ network.sent‚ 9.690425e+06 network.recv‚ 1.096265e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2694.78125 cpu‚26.851006782931194$ disk./.usageGB‚590.2887573242188 disk.dm-0.out‚5175.498046875 disk.sda.in‚0.109375 disk.rbd1.in‚0$ memory_percent‚99.19815063476562 disk.nvme0n1.in‚ 4558.28125 disk.dm-1.in‚0.109375 disk.sda3.out‚77.5625 disk.nvme0c0n1.in‚ 4558.28125 disk.dm-1.out‚77.5625* disk./.usagePercent‚39.614773789285074 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375) proc.memory.percent‚16.44763946533203 proc.cpu.threads‚17% disk.nvme0c0n1.out‚5175.498046875# disk.nvme0n1.out‚5175.498046875# proc.memory.availableMB‚131.375 disk.dm-0.in‚ 4558.28125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚77.5625 disk.rbd0.out‚0 disk.rbd1.out‚ 8.33203125¦y0+2à- _wandbruntime‚3418  _runtime‚3418øÅxÇáÁ trainer/global_step‚7249 % _timestamp‚1.7843714767881947e+09 _runtime‚3418.231975048 _step‚585 & lr-AdamW/pg1‚0.0007928278415030898 & lr-AdamW/pg2‚0.0007928278415030898˜nÓÌâÆ & lr-AdamW/pg1‚0.0007928278415030898 & lr-AdamW/pg2‚0.0007928278415030898 trainer/global_step‚7249 % _timestamp‚1.7843714767881947e+09 _runtime‚3418.231975048 _step‚585ÉÒ©a2ã- _wandbruntime‚3418  _runtime‚3418::­sväÍO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277712011595776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54466949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49931345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4423372449792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277712028635136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784371476789224e+09 _runtime‚3418.233211854 _step‚586 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265777796937728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4281440734208 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277312730370781 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281993364392960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54466951 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261354634084352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚7249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281993364392960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261354609180672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4266386569949 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273046326761472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4281440734208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33451665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936þÂý7Ø'åÒO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16479680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4266300432859 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54466951 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68844589 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281993364392960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54466951 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261354634084352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚7249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281993364392960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261354609180672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4266386569949 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273046326761472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4281440734208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33451665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277712011595776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54466949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49931345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4423372449792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277712028635136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209596416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784371476789224e+09 _runtime‚3418.233211854 _step‚586 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184692736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265777796937728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4281440734208 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277312730370781 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54474170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68853870 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4266867754389 _runtime‚ 3418.57304226 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211686400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277749139599360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54474172 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123338033 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54474170 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49937958 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186782720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4423767359488 trainer/global_step‚7249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123328040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123338033 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261389520695296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚587 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277749139599360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4281922156544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54474172 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54474172 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54474170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68853870 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4266867754389 _runtime‚ 3418.57304226 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211686400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277749139599360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54474172 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123338033 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54474170 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49937958 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186782720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4423767359488 trainer/global_step‚7249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123328040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123338033 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261389520695296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BIbatch_end/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273082798796800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282031044716544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚277749122560000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4282010100736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282031149700096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚587Ë«;2é- _wandbruntime‚3418  _runtime‚3418—½ êƒ 0loss/FRt2/assignment_loss‚0.6643956117851789 2loss/SRqqt2/assignment_loss‚0.5256764827184717 .loss/FBt2/detection_loss‚0.365599879035444 trainer/global_step‚7249 _runtime‚3418.574113774 2loss/SRqqt1/assignment_loss‚0.5942841550610393 3loss/FBt2/assignment_loss‚0.0008974920102651891 /loss/FBt1/detection_loss‚0.7985194281541875 %loss/total_loss‚8.347398136325682 epoch‚9 % _timestamp‚1.7843714771305778e+09 0loss/FRt1/assignment_loss‚1.6466286193865995 2loss/FBt1/assignment_loss‚6.53374101078035e-05 /loss/FRt1/detection_loss‚0.6370585891726268 1loss/SRqqt1/detection_loss‚0.8600472122295525 1loss/SRqqt2/detection_loss‚0.7205447733082627 /loss/FRt2/detection_loss‚1.5336805560539455 _step‚588Ú>çëˆ 0loss/FRt2/assignment_loss‚0.6643956117851789 2loss/SRqqt2/assignment_loss‚0.5256764827184717 .loss/FBt2/detection_loss‚0.365599879035444 trainer/global_step‚7249 _runtime‚3418.574113774 2loss/SRqqt1/assignment_loss‚0.5942841550610393 3loss/FBt2/assignment_loss‚0.0008974920102651891 /loss/FBt1/detection_loss‚0.7985194281541875 %loss/total_loss‚8.347398136325682 epoch‚9 % _timestamp‚1.7843714771305778e+09 0loss/FRt1/assignment_loss‚1.6466286193865995 2loss/FBt1/assignment_loss‚6.53374101078035e-05 /loss/FRt1/detection_loss‚0.6370585891726268 1loss/SRqqt1/detection_loss‚0.8600472122295525 1loss/SRqqt2/detection_loss‚0.7205447733082627 /loss/FRt2/detection_loss‚1.5336805560539455 _step‚588Ì°< ò‘ì‚(:† –²íÒéÐŒ gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Œ=rÓí‚(:È –²íÒóÔ‡) disk./.usagePercent‚39.61478454160151$ disk./.usageGB‚590.2889175415039 disk.rbd1.out‚ 8.33203125 disk.sda1.out‚0 disk.dm-0.out‚ 5182.4296875' proc.memory.availableMB‚ 131.4296875 network.recv‚ 1.100859e+06 disk.drbd2165.out‚0! disk.nvme0n1.out‚ 5182.4296875 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 77.8203125# proc.memory.rssMB‚ 2694.78515625 proc.cpu.threads‚17! disk.nvme0n1.in‚ 4558.37109375$ memory_percent‚99.19781684875488) proc.memory.percent‚16.44766330718994 disk.sda3.out‚ 77.8203125 disk.sda.in‚0.109375 disk.rbd1.in‚0# disk.nvme0c0n1.out‚ 5182.4296875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 4558.37109375 disk.sda2.out‚0 cpu‚26.848429780827633 disk.sda3.in‚0.109375 disk.sda.out‚ 77.8203125 disk.drbd2165.in‚0 disk.sda1.in‚0 network.sent‚ 9.747788e+06 disk.dm-0.in‚ 4558.37109375M«M&‘î‚(:† ¥²íÒù´¶Š gpu.0.gpu‚47.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yDµ·Ôï‚(:É ¥²íÒûÃŒŽ disk.dm-1.in‚0.109375$ memory_percent‚99.19829368591309# proc.memory.rssMB‚ 2694.78515625$ disk./.usageGB‚590.2891082763672 network.recv‚ 1.106036e+06# disk.nvme0n1.out‚5188.220703125 disk.rbd0.out‚0 disk.dm-1.out‚ 78.1015625 disk.dm-0.out‚5188.220703125) proc.memory.percent‚16.44766330718994 cpu‚26.845781244452496 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 4558.484375 disk.sda3.in‚0.109375 disk.sda3.out‚ 78.1015625 disk.sda.in‚0.109375% disk.nvme0c0n1.out‚5188.220703125 disk.nvme0n1.in‚ 4558.484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4558.484375' proc.memory.availableMB‚ 131.3515625 disk.rbd1.out‚ 8.39453125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 78.1015625 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 9.810478e+06* disk./.usagePercent‚39.614797341978225 disk.drbd2165.out‚0 disk.sda2.in‚0;ˆÛÐ2ð- _wandbruntime‚3440  _runtime‚3440£sEµÇñÁ _runtime‚3440.046955401 _step‚589 & lr-AdamW/pg1‚0.0007919874003567875 & lr-AdamW/pg2‚0.0007919874003567875 trainer/global_step‚7299 % _timestamp‚1.7843714986031466e+09‰™ÌòÆ _step‚589 & lr-AdamW/pg1‚0.0007919874003567875 & lr-AdamW/pg2‚0.0007919874003567875 trainer/global_step‚7299 % _timestamp‚1.7843714986031466e+09 _runtime‚3440.046955401ÍOÿBš2ó- _wandbruntime‚3440  _runtime‚3440ÆÒ½EÓ'ôÍO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274869928527872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚ 3440.04816795 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50263645 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4309853022208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4452621286400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4309853022208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263100209815552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283877412829184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69310491 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4309940959232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33675865 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4452793396224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283877517805568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16588027 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283877412829184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4294784549657 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4309940959232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279164730116889 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279567576846336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54828001 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54827999 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279164626940439 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4294698412567 trainer/global_step‚7299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16587780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283877517805568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267552806198272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279567559806976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50263895 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279567576846336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124148469¿EƒØ'õÒO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279567559806976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50263895 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279567576846336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124148469 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274869928527872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚ 3440.04816795 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50263645 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4309853022208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4452621286400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4309853022208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263100209815552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283877412829184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69310491 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4309940959232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33675865 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4452793396224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283877517805568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16588027 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283877412829184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4294784549657 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4309940959232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279164730116889 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279567576846336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54828001 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54827999 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279164626940439 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4294698412567 trainer/global_step‚7299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16587780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283877517805568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267552806198272ÎÓféE2ö- _wandbruntime‚3440  _runtime‚3440KˆFa÷ØM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _runtime‚3440.391790967 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69319772 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33680349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283915198136320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267588498790400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4295351871299 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124154992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54835222 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279604670771200 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124164985 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4310510325760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283915093152768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279604687810560 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4310510325760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69319772 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124164985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279604670771200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124154992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54835222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279201666297297 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267588305978368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚{už£}279604687810560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54835220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16590132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69329763 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50270241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283915198136320 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4310422381568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33680352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263135121330176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488ZxÏ„ã&øÝM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54835222 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279604670771200 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124164985 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4310510325760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283915093152768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279604687810560 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4310510325760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69319772 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124164985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279604670771200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124154992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54835222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279201666297297 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267588305978368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279604687810560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54835220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16590132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69329763 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50270241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283915198136320 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4310422381568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33680352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263135121330176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _runtime‚3440.391790967 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69319772 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33680349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283915198136320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267588498790400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4295351871299 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124154992Ï6Êdþ2ù- _wandbruntime‚3440  _runtime‚3440¹ÌV ú‚ 0loss/FRt2/assignment_loss‚0.8234549350854827 2loss/SRqqt2/assignment_loss‚0.5220523216391207 /loss/FRt1/detection_loss‚0.6026271141211073 .loss/FRt2/detection_loss‚1.500468168571228 .loss/FBt1/detection_loss‚0.897442299489151 %loss/total_loss‚8.661421029040966 _step‚592 2loss/FBt2/assignment_loss‚0.005238707907727487 2loss/SRqqt1/assignment_loss‚0.6228494287557934 4loss/FBt1/assignment_loss‚0.00029468176405637625 1loss/SRqqt1/detection_loss‚0.9672644739919456 1loss/SRqqt2/detection_loss‚0.5759646728569267 epoch‚9 _runtime‚3440.392936258 0loss/FRt1/assignment_loss‚1.6812622154087533 /loss/FBt2/detection_loss‚0.4625020094496745 trainer/global_step‚7299 $ _timestamp‚1.784371498949395e+09¬'Žµ û‡ epoch‚9 _runtime‚3440.392936258 0loss/FRt1/assignment_loss‚1.6812622154087533 /loss/FBt2/detection_loss‚0.4625020094496745 trainer/global_step‚7299 $ _timestamp‚1.784371498949395e+09 0loss/FRt2/assignment_loss‚0.8234549350854827 2loss/SRqqt2/assignment_loss‚0.5220523216391207 /loss/FRt1/detection_loss‚0.6026271141211073 .loss/FRt2/detection_loss‚1.500468168571228 .loss/FBt1/detection_loss‚0.897442299489151 %loss/total_loss‚8.661421029040966 _step‚592 2loss/FBt2/assignment_loss‚0.005238707907727487 2loss/SRqqt1/assignment_loss‚0.6228494287557934 4loss/FBt1/assignment_loss‚0.00029468176405637625 1loss/SRqqt1/detection_loss‚0.9672644739919456 1loss/SRqqt2/detection_loss‚0.5759646728569267З†Ó‘ü‚(:† ´²íÒáÿëŠ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚179.098' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pöLÒáý‚(:Ö ´²íÒ½ÔöŒ network.sent‚ 9.815357e+06 disk.sda.in‚0.109375 disk.sda.out‚ 78.45703125# proc.memory.rssMB‚ 2694.78515625 disk.sda1.out‚0! disk.dm-0.out‚5191.3955078125$ memory_percent‚99.20194149017334$ disk./.usageGB‚590.2902183532715 disk.sda1.in‚0 disk.dm-0.in‚ 4558.48828125( proc.memory.availableMB‚ 130.75390625 disk.rbd1.in‚0) proc.memory.percent‚16.44766330718994* disk./.usagePercent‚39.614871840170686 disk.dm-1.in‚0.109375 disk.sda3.out‚ 78.45703125 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 1.109219e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 78.45703125 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.out‚ 8.39453125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4558.48828125& disk.nvme0c0n1.out‚5191.3955078125! disk.nvme0n1.in‚ 4558.48828125$ disk.nvme0n1.out‚5191.3955078125 cpu‚26.843157915866357’<2þ- _wandbruntime‚3462  _runtime‚3462YÜ(²ÇÿÁ % _timestamp‚1.7843715206605253e+09 _runtime‚3462.104313061 _step‚593 & lr-AdamW/pg1‚0.0007911410656316089 & lr-AdamW/pg2‚0.0007911410656316089 trainer/global_step‚7349ÊWWÅÌ€Æ & lr-AdamW/pg1‚0.0007911410656316089 & lr-AdamW/pg2‚0.0007911410656316089 trainer/global_step‚7349 % _timestamp‚1.7843715206605253e+09 _runtime‚3462.104313061 _step‚593Ñâÿ¾–2- _wandbruntime‚3462  _runtime‚3462†uXÓ'‚ÍO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16696721 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269327994639360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784371520661635e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚594 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55189049 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264845760643072 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4338441184256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285761461265408 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285761461265408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4323096392275 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚281423125057536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124975421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124965442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33900068 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33900065 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4482032788480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4338353247232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281016626686547 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276693547333632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269327793431552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4323182529365 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚281423108018176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16696954 _runtime‚3462.105625598ÍIٯƒÒO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4338441184256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285761461265408 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285761461265408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4323096392275 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚281423125057536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124975421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124965442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33900068 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33900065 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4482032788480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4338353247232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281016626686547 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276693547333632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269327793431552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4323182529365 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚281423108018176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16696954 _runtime‚3462.105625598 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16696721 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269327994639360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784371520661635e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚594 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55189049 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264845760643072Ò~fI2„- _wandbruntime‚3462  _runtime‚3462¡†pÝ&…×M LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 trainer/global_step‚7349 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4338922606592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚281460236021760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124981944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285799141588992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264880672157696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4482794958848 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50603328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚595 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4323663713805 _runtime‚3462.447360467 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4338922606592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69795665 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269363297111040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276730019368960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50603548 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55196270 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69785674 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784371521003312e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚281460236021760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281053666043405 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55196270 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16698779 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69785674 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285799246572544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33904549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124981944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482624953344 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264880697061376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55196272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4339010550784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4339010550784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124991937 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281053769219967¯GYzâ&†ÜM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33904549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124981944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482624953344 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264880697061376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55196272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4339010550784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4339010550784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124991937 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281053769219967 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 trainer/global_step‚7349 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4338922606592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚281460236021760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124981944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285799141588992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264880672157696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4482794958848 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50603328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚595 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4323663713805 _runtime‚3462.447360467 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4338922606592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69795665 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269363297111040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276730019368960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50603548 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55196270 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69785674 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784371521003312e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚281460236021760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281053666043405 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55196270 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16698779 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69785674 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285799246572544Ó Œ2‡- _wandbruntime‚3462  _runtime‚3462<Åï ˆƒ 0loss/FBt2/detection_loss‚0.30751953735986415 2loss/SRqqt2/assignment_loss‚0.5213089466690763 3loss/FBt1/assignment_loss‚0.0014925057606996522 1loss/FBt2/assignment_loss‚0.12516189431238717 /loss/FBt1/detection_loss‚0.5444656277752735 epoch‚9 % _timestamp‚1.7843715210049968e+09 _runtime‚3462.448555793 2loss/SRqqt1/assignment_loss‚0.6130963474914635 /loss/FRt2/detection_loss‚1.6408562830300721 1loss/SRqqt2/detection_loss‚0.5978304799646139 0loss/FRt2/assignment_loss‚0.6714252027053407 .loss/FRt1/detection_loss‚0.492488125198794 %loss/total_loss‚8.235019286660144 trainer/global_step‚7349 _step‚596 0loss/FRt1/assignment_loss‚1.7566936773193234 1loss/SRqqt1/detection_loss‚0.9626806590732344? Ññ‰ˆ 0loss/FRt2/assignment_loss‚0.6714252027053407 .loss/FRt1/detection_loss‚0.492488125198794 %loss/total_loss‚8.235019286660144 trainer/global_step‚7349 _step‚596 0loss/FRt1/assignment_loss‚1.7566936773193234 1loss/SRqqt1/detection_loss‚0.9626806590732344 0loss/FBt2/detection_loss‚0.30751953735986415 2loss/SRqqt2/assignment_loss‚0.5213089466690763 3loss/FBt1/assignment_loss‚0.0014925057606996522 1loss/FBt2/assignment_loss‚0.12516189431238717 /loss/FBt1/detection_loss‚0.5444656277752735 epoch‚9 % _timestamp‚1.7843715210049968e+09 _runtime‚3462.448555793 2loss/SRqqt1/assignment_loss‚0.6130963474914635 /loss/FRt2/detection_loss‚1.6408562830300721 1loss/SRqqt2/detection_loss‚0.5978304799646139Ô}g ‘Š‚(:† òíҽǮ‹ gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚172.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨¸¤×‹‚(:Ì òíÒƒ‹º disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 4637.12109375$ memory_percent‚99.20251369476318 disk.rbd1.out‚ 8.45703125& disk.nvme0c0n1.out‚5196.8798828125 disk.sda3.in‚0.109375 network.sent‚ 9.878159e+06* disk./.usagePercent‚39.614803230151516$ disk./.usageGB‚590.2891960144043 disk.sda1.in‚0 disk.dm-1.in‚0.109375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2694.78515625) proc.memory.percent‚16.44766330718994 disk.sda2.in‚0! disk.nvme0n1.in‚ 4637.12109375( proc.memory.availableMB‚ 130.66015625! disk.dm-0.out‚5196.8798828125 disk.rbd0.out‚0 disk.sda3.out‚78.71875 disk.dm-1.out‚78.71875 disk.sda.out‚78.71875$ disk.nvme0n1.out‚5196.8798828125 cpu‚26.840530519901375 proc.cpu.threads‚17 disk.dm-0.in‚ 4637.12109375 network.recv‚ 1.11447e+06 disk.sda2.out‚0 disk.sda.in‚0.109375šÑ¦,‘Œ‚(:† Ò²íҴऋ gpu.0.gpu‚61.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞC¯ Û‚(:Ð Ò²íÒÌáž disk.dm-1.out‚ 78.88671875# proc.memory.rssMB‚ 2694.78515625) disk./.usagePercent‚39.61479606194055$ disk./.usageGB‚590.2890892028809 disk.drbd2165.out‚0 disk.rbd1.out‚ 8.45703125 disk.nvme0n1.in‚ 4637.3984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375& proc.memory.availableMB‚ 130.859375 disk.dm-0.in‚ 4637.3984375! disk.dm-0.out‚5201.8603515625 disk.sda3.out‚ 78.88671875 disk.sda.in‚0.109375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.44766330718994 cpu‚26.838151092044978& disk.nvme0c0n1.out‚5201.8603515625 disk.sda.out‚ 78.88671875$ disk.nvme0n1.out‚5201.8603515625 disk.sda3.in‚0.109375 proc.cpu.threads‚17 network.sent‚ 9.941022e+06" disk.nvme0c0n1.in‚ 4637.3984375$ memory_percent‚99.20129776000977 disk.rbd0.in‚0 network.recv‚ 1.119428e+06 disk.drbd2165.in‚0éyr2Ž- _wandbruntime‚3483  _runtime‚3483ˆQr,ÇÁ & lr-AdamW/pg1‚0.0007902888508833222 & lr-AdamW/pg2‚0.0007902888508833222 trainer/global_step‚7399 % _timestamp‚1.7843715424032016e+09 _runtime‚3483.847009724 _step‚597ѦÌáÌÆ trainer/global_step‚7399 % _timestamp‚1.7843715424032016e+09 _runtime‚3483.847009724 _step‚597 & lr-AdamW/pg1‚0.0007902888508833222 & lr-AdamW/pg2‚0.0007902888508833222Õq -Q2‘- _wandbruntime‚3483  _runtime‚3483+‘¹Eu ’ÎO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287645509701632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55550099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4366941409280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16804765 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278517149100032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34124265 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4511504388608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50929278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266591336374272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 _runtime‚3483.848234956 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55550099 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283278673268736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271102840762880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346øc“ Ù'“ÓO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70242295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34124268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282868626432655 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70242295 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚287645614678016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287645614678016 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4511504388608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50929278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266591336374272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 _runtime‚3483.848234956 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55550099 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283278673268736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271102840762880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287645509701632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55550099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4366941409280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16804765 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278517149100032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34124265 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50935758 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843715427484946e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283315784232960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16807246 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16807009 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278553621135360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4352061693513 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4367510775808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266626272792576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282905665789513 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125808896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70251576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4512266317312 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4367510775808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70261567 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278553604096000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287683190025216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚287683295008768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4512100506112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚287683295008768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283315784232960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271138539109888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271138348395008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70261567 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55557322 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125818889 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34128752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70251576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266626247888896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165811200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283315767193600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237¦Ë‹‚•–ÝM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4367510775808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70261567 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278553604096000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287683190025216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚287683295008768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4512100506112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚287683295008768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283315784232960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271138539109888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271138348395008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70261567 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55557322 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125818889 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34128752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70251576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266626247888896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165811200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283315767193600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55557322 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50935758 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843715427484946e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283315784232960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16807246 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16807009 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278553621135360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4352061693513 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4367510775808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266626272792576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282905665789513 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125808896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70251576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4512266317312 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289529558137856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285134221479936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70708197 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280340750866432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379892351691 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51262177 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285134204440576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚ 3505.91579816 _step‚602 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55911149 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843715644717462e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4395353697280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272878086781952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284720626178763 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379978488781 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70708197 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51261931 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289529663114240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16913709 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280340733827072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34348468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55911151 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285134221479936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395441634304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70718174 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126629325 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚ 3505.91579816 _step‚602 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55911149 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843715644717462e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4395353697280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272878086781952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284720626178763 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379978488781 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70708197 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51261931 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289529663114240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16913709 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280340733827072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34348468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55911151 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285134221479936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395441634304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70718174 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126629325 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289529558137856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285134221479936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70708197ÚÚµ$É2¢- _wandbruntime‚3506  _runtime‚35065? úÇ£ÕM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396011000832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272913379312640 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4541736183808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚7449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55918372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126645841 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34352952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16915899 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272913584707584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70727469 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocatÉ–²ôed‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289567343444992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55918372 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55918370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70717478 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126645841 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126635848 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51268597 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34352949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284757665535621 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70727469 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396011000832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268371823620096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4395923056640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285171315404800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285171332444160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51268851 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4380459673221 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289567343444992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126635848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4541555692544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4395923056640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289567238461440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280377205862400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚7449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55918372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126645841 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34352952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16915899 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272913584707584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70727469 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289567343444992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55918372 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55918370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70717478 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126645841 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126635848 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51268597 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34352949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284757665535621 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70727469 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396011000832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268371823620096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4395923056640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285171315404800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285171332444160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51268851 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4380459673221 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289567343444992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126635848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4541555692544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4395923056640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289567238461440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280377205862400 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4541736183808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274653104202752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127446298 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71184076 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4423853922304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56272201 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291413606574080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34572668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4408290331399 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56272199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286989769691136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71174099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4423853922304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 trainer/global_step‚7499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34572665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286989752651776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127456277 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51594303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4570419352576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286989769691136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚3527.883614398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286989752651776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17021635 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327bø£Á±ÒO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4423853922304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56272201 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291413606574080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34572668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4408290331399 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56272199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286989769691136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71174099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4423853922304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 trainer/global_step‚7499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34572665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286989752651776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127456277 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51594303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4570419352576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286989769691136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚3527.883614398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286989752651776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17021635 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71174099 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270082487836672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274653104202752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127446298 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71184076 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144Þ8l)y2²- _wandbruntime‚3528  _runtime‚3528µí|·Þ&³ØM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4424423281664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4408857652929 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4424511225856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4408943790131 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282200824668160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291451391881216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56279422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274688601452544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71193371 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4571177197568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843715867787006e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34577152 trainer/global_step‚7499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71193371 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71183380 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34577149 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286609665281729 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4424511225856 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287026880655360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51600949 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127472793 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51600723 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287026880655360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56279420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270117399351296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3528.222779783 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56279422 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127462800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270117424254976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4571007192064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287026863616000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287026863616000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127472793 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚282200807628800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127462800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17023574 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56279420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17023797€õã&´ÝM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71193371 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4571177197568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843715867787006e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34577152 trainer/global_step‚7499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71193371 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71183380 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34577149 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286609665281729 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4424511225856 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287026880655360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51600949 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127472793 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51600723 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287026880655360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56279420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270117399351296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3528.222779783 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56279422 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127462800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270117424254976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4571007192064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 _step‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287026863616000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287026863616000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127472793 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚282200807628800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127462800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17023574 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56279420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17023797 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4424423281664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4408857652929 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4424511225856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4408943790131 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282200824668160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291451391881216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56279422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274688601452544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192ßíCc2µ- _wandbruntime‚3528  _runtime‚3528¨å½¶€ /loss/FRt2/detection_loss‚1.6738234287857807 0loss/SRqqt2/detection_loss‚0.672183741863979 /loss/FBt1/detection_loss‚0.9876209421302465 epoch‚9 trainer/global_step‚7499 2loss/SRqqt1/assignment_loss‚0.5224956764646969 2loss/SRqqt2/assignment_loss‚0.6151941957420486 3loss/FBt1/assignment_loss‚0.0024232264580966918 1loss/SRqqt1/detection_loss‚0.9044992833196656 _runtime‚3528.223798586 _step‚608 0loss/FRt1/assignment_loss‚1.6843112592876193 # _timestamp‚1.78437158678027e+09 %loss/total_loss‚9.250258091875352 0loss/FRt2/assignment_loss‚0.8525721179472433 1loss/FBt2/assignment_loss‚0.00866352636558885 /loss/FRt1/detection_loss‚0.6302527946488834 /loss/FBt2/detection_loss‚0.6962178988615051dZ> ·… 1loss/SRqqt1/detection_loss‚0.9044992833196656 _runtime‚3528.223798586 _step‚608 0loss/FRt1/assignment_loss‚1.6843112592876193 # _timestamp‚1.78437158678027e+09 %loss/total_loss‚9.250258091875352 0loss/FRt2/assignment_loss‚0.8525721179472433 1loss/FBt2/assignment_loss‚0.00866352636558885 /loss/FRt1/detection_loss‚0.6302527946488834 /loss/FBt2/detection_loss‚0.6962178988615051 /loss/FRt2/detection_loss‚1.6738234287857807 0loss/SRqqt2/detection_loss‚0.672183741863979 /loss/FBt1/detection_loss‚0.9876209421302465 epoch‚9 trainer/global_step‚7499 2loss/SRqqt1/assignment_loss‚0.5224956764646969 2loss/SRqqt2/assignment_loss‚0.6151941957420486 3loss/FBt1/assignment_loss‚0.0024232264580966918à‡!›‘¸‚(:† Ž³íÒêæ’‹ gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚189.042' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚034F»Þ¹‚(:Ó Ž³íÒí—ÈŒ disk.drbd2165.out‚0& disk.nvme0c0n1.out‚5226.4306640625! disk.nvme0n1.in‚ 4714.18359375$ memory_percent‚99.20408725738525 network.sent‚ 1.0076271e+07$ disk./.usageGB‚590.2901878356934 disk.drbd2165.in‚0 disk.rbd1.out‚ 8.58203125# disk.nvme0c0n1.in‚ 4714.18359375 disk.sda1.in‚0 disk.dm-0.in‚ 4714.18359375 disk.sda3.in‚0.109375 disk.rbd1.in‚0$ disk.nvme0n1.out‚5226.4306640625) disk./.usagePercent‚39.61486979211041 disk.sda3.out‚ 80.1015625# proc.memory.rssMB‚ 2694.78515625 cpu‚26.828060212256215 disk.sda1.out‚0( proc.memory.availableMB‚ 130.40234375) proc.memory.percent‚16.44766330718994 network.recv‚ 1.136128e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 80.1015625 disk.sda.in‚0.109375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0.109375! disk.dm-0.out‚5226.4306640625 disk.rbd0.in‚0 disk.sda.out‚ 80.1015625¶¨Éj2º- _wandbruntime‚3549  _runtime‚3549V`§¬Ç»Á & lr-AdamW/pg1‚0.0007876970634688938 & lr-AdamW/pg2‚0.0007876970634688938 trainer/global_step‚7549 % _timestamp‚1.7843716083074958e+09 _runtime‚3549.751274723 _step‚609¤dÌ¼Æ trainer/global_step‚7549 % _timestamp‚1.7843716083074958e+09 _runtime‚3549.751274723 _step‚609 & lr-AdamW/pg1‚0.0007876970634688938 & lr-AdamW/pg2‚0.0007876970634688938áMi£p2½- _wandbruntime‚3549  _runtime‚3549weŠQë¾ÎO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288845300862976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17130224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.yïi,é%small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283987954399232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56633251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56633249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71640001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34796868 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4600051885056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276428140356608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71649978 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288845317902336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4452354147328 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17129982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288424625670979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283987937359872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436774448197 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34796865 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271828063567872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 trainer/global_step‚7549 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71640001 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288845300862976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71649978 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4436688311107 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚610 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288845317902336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293297655010304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276427943343104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271828088471552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293297655010304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128283229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288424728847429 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128273250 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473ï;¹ÔÙ'¿ÓO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288845300862976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17130224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283987954399232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56633251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56633249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71640001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34796868 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4600051885056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276428140356608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71649978 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288845317902336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4452354147328 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17129982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288424625670979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283987937359872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436774448197 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34796865 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271828063567872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 trainer/global_step‚7549 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71640001 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288845300862976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71649978 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4436688311107 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚610 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288845317902336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293297655010304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276427943343104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271828088471552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293297655010304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128283229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288424728847429 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128273250 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360â±c“t2À- _wandbruntime‚3550  _runtime‚3550”MNÞ&ÁØM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293335440317440 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271862999986176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71649282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34801349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71659273 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71659273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288882411827200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4453011450880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271862975082496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17132365 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288882428866560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4452923506688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚3550.096571132 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51933486 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128299745 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚284024409395200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128289752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288882428866560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284024426434560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71649282 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4437341769839 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4452923506688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17132137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288461665027837 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293335335333888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640Á‚ÜX ÂÝM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56640472 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4437341769839 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_uöƒkØend/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4452923506688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17132137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288461665027837 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293335335333888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293335440317440 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271862999986176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71649282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34801349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71659273 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71659273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288882411827200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4453011450880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271862975082496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17132365 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288882428866560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4452923506688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚3550.096571132 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51933486 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128299745 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚284024409395200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128289752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288882428866560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284024426434560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71649282 F2Ã- _wandbruntime‚3550  _runtime‚3550·ªâÄ 1loss/SRqqt1/detection_loss‚0.8517124040319802 1loss/SRqqt2/detection_loss‚0.6449003560719406 epoch‚9 /loss/FBt1/detection_loss‚0.4818054159956151 _runtime‚3550.097814068 2loss/SRqqt2/assignment_loss‚0.5620718844767102 1loss/FBt1/assignment_loss‚0.01727484098460994 .loss/FBt2/detection_loss‚0.495437215931975 %loss/total_loss‚8.254329079409862 0loss/FRt1/assignment_loss‚1.6943520314275882 0loss/FRt2/assignment_loss‚0.7423301333499746 trainer/global_step‚7549 % _timestamp‚1.7843716086542614e+09 _step‚612 2loss/SRqqt1/assignment_loss‚0.5856230053324636 2loss/FBt2/assignment_loss‚0.006352198014774295 /loss/FRt1/detection_loss‚0.5700742796427782 /loss/FRt2/detection_loss‚1.6023953141494527Y™~ ņ 0loss/FRt2/assignment_loss‚0.7423301333499746 trainer/global_step‚7549 % _timestamp‚1.7843716086542614e+09 _step‚612 2loss/SRqqt1/assignment_loss‚0.5856230053324636 2loss/FBt2/assignment_loss‚0.006352198014774295 /loss/FRt1/detection_loss‚0.5700742796427782 /loss/FRt2/detection_loss‚1.6023953141494527 1loss/SRqqt1/detection_loss‚0.8517124040319802 1loss/SRqqt2/detection_loss‚0.6449003560719406 epoch‚9 /loss/FBt1/detection_loss‚0.4818054159956151 _runtime‚3550.097814068 2loss/SRqqt2/assignment_loss‚0.5620718844767102 1loss/FBt1/assignment_loss‚0.01727484098460994 .loss/FBt2/detection_loss‚0.495437215931975 %loss/total_loss‚8.254329079409862 0loss/FRt1/assignment_loss‚1.6943520314275882ä3Ñ0âÆ‚(:… ³íҤĈ‹ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚176.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·íâÇ‚(:× ³í҃߅Ž disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 8.64453125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 80.31640625 disk.rbd0.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 4714.18359375# memory_percent‚99.2051362991333( proc.memory.availableMB‚ 130.23046875$ disk./.usageGB‚590.2894668579102 disk.sda.in‚0.109375 disk.rbd1.in‚0 network.sent‚ 1.0138966e+07 disk.sda2.out‚0 disk.dm-0.in‚ 4714.18359375 disk.sda3.in‚0.109375' disk.nvme0c0n1.out‚5232.79150390625 network.recv‚ 1.141585e+06 cpu‚26.82554707618571# disk.nvme0c0n1.in‚ 4714.18359375% disk.nvme0n1.out‚5232.79150390625# proc.memory.rssMB‚ 2694.78515625) proc.memory.percent‚16.44766330718994 disk.rbd0.out‚0) disk./.usagePercent‚39.61482140668644 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 80.31640625" disk.dm-0.out‚5232.79150390625 disk.sda.out‚ 80.31640625…5¸›È‚(: ¬³íÒ µÖŠ gpu.0.gpu‚56.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚182.63' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.525999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,\0ÙÉ‚(:Î ¬³íÒÃÄúŒ disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 80.6953125 disk.sda.out‚ 80.6953125# disk.nvme0c0n1.in‚ 4714.18359375 network.recv‚ 1.146736e+06 disk.sda3.out‚ 80.6953125 network.sent‚ 1.020158e+07! disk.nvme0n1.in‚ 4714.18359375 disk.rbd0.out‚0 disk.dm-0.out‚5236.150390625 disk.drbd2165.in‚0$ memory_percent‚99.20287132263184 proc.cpu.threads‚17# disk.nvme0n1.out‚5236.150390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375 disk.rbd1.out‚ 8.64453125 disk.sda1.in‚0 cpu‚26.823094828484148$ disk./.usageGB‚590.2905006408691 disk.drbd2165.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚5236.150390625' proc.memory.availableMB‚ 130.6015625# proc.memory.rssMB‚ 2694.78515625 disk.dm-0.in‚ 4714.18359375) disk./.usagePercent‚39.61489078472822 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375) proc.memory.percent‚16.44766330718994ç>'Ô2Ê- _wandbruntime‚3571  _runtime‚3571+CqÄÇËÁ trainer/global_step‚7599 % _timestamp‚1.7843716303339717e+09 _runtime‚3571.777769568 _step‚613 & lr-AdamW/pg1‚0.0007868214660661819 & lr-AdamW/pg2‚0.0007868214660661819nÐÉÌÌÆ & lr-AdamW/pg1‚0.0007868214660661819 & lr-AdamW/pg2‚0.0007868214660661819 trainer/global_step‚7599 % _timestamp‚1.7843716303339717e+09 _runtime‚3571.777769568 _step‚613åÿMÎ2Í- _wandbruntime‚3571  _runtime‚3571D“ðôÔ'ÎÎO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129100202 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4465086290815 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52259913 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4629347863552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4480854372352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4480942309376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4480854372352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290700849074176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚273573664202752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72105903 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35021065 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843716303350837e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285811539126272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35021068 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17239078 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4629522070528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72105903 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56994301 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295181703446528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4480942309376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290700866113536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273573639299072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290700866113536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129110181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295181703446528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290700849074176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17238848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295181808422912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 trainer/global_step‚7599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56994299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _runtime‚3571.779056688 _step‚614 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52260146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290276625417087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚295181808422912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56994301 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285811556165632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450œ,ÜVÙ'ÏÓO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129110181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295181703446528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290700849074176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17238848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295181808422912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 trainer/global_step‚7599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56994299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _runtime‚3571.779056688 _step‚614 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52260146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290276625417087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚295181808422912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56994301 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285811556165632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129100202 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4465086290815 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52259913 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4629347863552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4480854372352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4480942309376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4480854372352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290700849074176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚273573664202752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72105903 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35021065 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843716303350837e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285811539126272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35021068 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17239078 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4629522070528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72105903 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56994301 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295181703446528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4480942309376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290700866113536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273573639299072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290700866113536æ“Ë2Ð- _wandbruntime‚3572  _runtime‚3572è(üØ·Ñ×M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 L=DeviݪÛ&&ceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295219488753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290737960038400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52266766 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚7599 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57001522 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290313767950507 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129116704 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35025549 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72115184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4481423731712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129126697 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3572.119036961 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4481511675904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17240993 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295219488753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4629937413120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72125175 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4481511675904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17241214 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57001522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72115184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285848011161600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57001520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52266542 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273608550813696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295219383770112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4481423731712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273608575717376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35025552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290737977077760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290313664773945 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4465739749547 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290737977077760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4465653612345 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784371630674877e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290737960038400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4630111612928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129126697 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285848028200960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146®€åâ&ÒÜM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52266542 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273608550813696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295219383770112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4481423731712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273608575717376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35025552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290737977077760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290313664773945 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4465739749547 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290737977077760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4465653612345 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784371630674877e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290737960038400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4630111612928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129126697 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285848028200960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295219488753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290737960038400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52266766 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚7599 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57001522 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290313767950507 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _step‚615 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129116704 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35025549 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72115184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4481423731712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129126697 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3572.119036961 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4481511675904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17240993 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295219488753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4629937413120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72125175 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4481511675904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17241214 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57001522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72115184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285848011161600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57001520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297065751882752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4659015831040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4493484270523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚223 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292556397285376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279978060848640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 $ _timestamp‚1.784371652170654e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275319239933952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287635140892672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35245265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3593.617579646 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129927154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4658845818368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57355349 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292128728339645 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72571805 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚618 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4509442534400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17347394 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297065856859136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4509442534400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509354597376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52592436 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275319215030272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292128625163195 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297065751882752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292556414324736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292556397285376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129937133 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4493570407613 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509354597376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279978255764992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚7649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297065856859136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57355351 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17347171 ZKDeviceStatsMonitor.on_train_batch_start/reservedé)Ú_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57355351 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72581782 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146•©¦ÅØ'ÝÒO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72581782 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72571805 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52592662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297065751882752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4659015831040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4493484270523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚223 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292556397285376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279978060848640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 $ _timestamp‚1.784371652170654e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275319239933952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287635140892672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35245265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3593.617579646 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129927154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4658845818368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57355349 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292128728339645 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72571805 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚618 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4509442534400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17347394 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297065856859136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4509442534400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509354597376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52592436 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275319215030272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292128625163195 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297065751882752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292556414324736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292556397285376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129937133 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4493570407613 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509354597376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279978255764992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚7649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297065856859136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57355351 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17347171 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57355351 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360ê‹bçu2Þ- _wandbruntime‚3593  _runtime‚3593û×Þ&ßØM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72591077 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57362572 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52599069 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57362570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275354126544896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297103432206336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129943656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17349320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129953649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843716525129817e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129953649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292593525288960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4510011900928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4659606092288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297103537189888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280013757540864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297103537189888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292165767696615 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275354151448576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292593525288960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35249749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129943656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280013564728832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4510011900928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292165664520053 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17349556 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72581086 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52599069 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57362570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275354126544896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297103432206336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129943656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17349320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129953649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 % _timestamp‚1.7843716525129817e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129953649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292593525288960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4510011900928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4659606092288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297103537189888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280013757540864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297103537189888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292165767696615 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275354151448576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292593525288960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35249749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129943656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280013564728832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4510011900928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292165664520053 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17349556 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72581086 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298949800318976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚289458759698432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 $ _timestamp‚1.784371674266712e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294411945496576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294411962535936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73037707 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4688296687616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289458742659072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277064815665152 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73037707 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298949905295360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294411962535936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537854822400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17455691 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281753087449088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4521968387321 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4537942759424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35469465 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57716401 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4688479283200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73047684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294411945496576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298949800318976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277064790761472=øNØ'íÒO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52924926 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73037707 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4688296687616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289458742659072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277064815665152 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73037707 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298949905295360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294411962535936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537854822400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17455691 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281753087449088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4521968387321 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4537942759424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35469465 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57716401 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4688479283200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73047684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294411945496576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298949800318976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277064790761472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298949800318976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚289458759698432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 $ _timestamp‚1.784371674266712e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294411945496576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294411962535936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130780601 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚3616.052536963 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73056979 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4689063433216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298987480642560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277099727179776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294449056460800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35473952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538424181760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52931843 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73046988 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298987585626112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130770608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57723622 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130780601 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130770608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚289495214694400 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52931595 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 % _timestamp‚1.7843716746083708e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4538424181760 \JDeviceStatsMonitor.on_train_batch_end/allocated_beÔ¶¿ytes.small_pool.allocated‚ 4538512125952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294017664266161 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _step‚623 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57723620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281788790612992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0œ“Øaã&ðÝM BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52931595 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 % _timestamp‚1.7843716746083708e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4538424181760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4538512125952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294017664266161 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _step‚623 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57723620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281788790612992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57723622 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277099702276096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294017767442723 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4522535708963 trainer/global_step‚7699 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73046988 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294449073500160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4688882941952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294449056460800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289495231733760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130780601 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚3616.052536963 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73056979 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4689063433216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298987480642560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277099727179776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294449056460800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35473952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538424181760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52931843 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73046988 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298987585626112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130770608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57723622 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130780601 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130770608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚289495214694400ï"Ä‘©2ñ- _wandbruntime‚3616  _runtime‚3616ãV¼ò‚ _runtime‚3616.053603687 0loss/FRt1/assignment_loss‚1.6224387992830838 2loss/SRqqt1/assignment_loss‚0.5912548239187648 2loss/FBt1/assignment_loss‚5.73749417321284e-06 epoch‚9 2loss/SRqqt2/assignment_loss‚0.4713012660715385 1loss/SRqqt2/detection_loss‚0.6461758976761501 % _timestamp‚1.7843716746100478e+09 0loss/FRt2/assignment_loss‚0.6932792749490824 /loss/FRt2/detection_loss‚1.4917625108616308 0loss/SRqqt1/detection_loss‚0.900080435053115 _step‚624 1loss/FBt2/assignment_loss‚0.00960760980342593 /loss/FRt1/detection_loss‚0.5811099689001732 0loss/FBt1/detection_loss‚0.32578320971325686 /loss/FBt2/detection_loss‚0.6019393969486685 %loss/total_loss‚7.934738930673063 trainer/global_step‚7699œY†n ó‡ 0loss/FRt2/assignment_loss‚0.6932792749490824 /loss/FRt2/detection_loss‚1.4917625108616308 0loss/SRqqt1/detection_loss‚0.900080435053115 _step‚624 1loss/FBt2/assignment_loss‚0.00960760980342593 /loss/FRt1/detection_loss‚0.5811099689001732 0loss/FBt1/detection_loss‚0.32578320971325686 /loss/FBt2/detection_loss‚0.6019393969486685 %loss/total_loss‚7.934738930673063 trainer/global_step‚7699 _runtime‚3616.053603687 0loss/FRt1/assignment_loss‚1.6224387992830838 2loss/SRqqt1/assignment_loss‚0.5912548239187648 2loss/FBt1/assignment_loss‚5.73749417321284e-06 epoch‚9 2loss/SRqqt2/assignment_loss‚0.4713012660715385 1loss/SRqqt2/detection_loss‚0.6461758976761501 % _timestamp‚1.7843716746100478e+09ðŠ„çN‘ô‚(:† è³íÒîó‹ gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚181.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ϭ<Üâõ‚(:× è³íÒÿë‚ disk.rbd0.out‚0 disk.sda.in‚0.109375 cpu‚26.81366270886616 network.sent‚ 1.034485e+07 disk.sda1.in‚0 disk.dm-1.out‚ 81.76953125 disk.sda3.in‚0.109375 disk.rbd1.out‚ 8.76953125' disk.nvme0c0n1.out‚5252.68701171875 network.recv‚ 1.178659e+06 disk.sda2.out‚0 disk.dm-0.in‚ 4818.62109375 disk.drbd2165.out‚0 disk.rbd1.in‚0) proc.memory.percent‚16.44766330718994$ disk./.usageGB‚590.2907943725586 disk.sda3.out‚ 81.76953125$ memory_percent‚99.20556545257568 proc.cpu.threads‚17 disk.sda.out‚ 81.76953125 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2694.78515625! disk.nvme0n1.in‚ 4818.62109375% disk.nvme0n1.out‚5252.68701171875 disk.sda2.in‚0 disk.dm-1.in‚0.109375) disk./.usagePercent‚39.61491049730836 disk.rbd0.in‚0 disk.sda1.out‚0" disk.dm-0.out‚5252.68701171875# disk.nvme0c0n1.in‚ 4818.62109375( proc.memory.availableMB‚ 130.16015625ömiw2ö- _wandbruntime‚3637  _runtime‚3637c%u Ã÷½ & lr-AdamW/pg1‚0.0007841598654053411 & lr-AdamW/pg2‚0.0007841598654053411 trainer/global_step‚7749 ! _timestamp‚1.784371696086e+09 _runtime‚3637.529873811 _step‚625NL¨mÈø ! _timestamp‚1.784371696086e+09 _runtime‚3637.529873811 _step‚625 & lr-AdamW/pg1‚0.0007841598654053411 & lr-AdamW/pg2‚0.0007841598654053411 trainer/global_step‚7749ñÁäÌ2ù- _wandbruntime‚3637  _runtime‚3637ñþ8ðÔ'úÎO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291282361464832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73503609 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278810391396352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3637.534582862 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300833848755200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296267510747136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4566442984448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296267493707776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291282344425472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚7749 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53257704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35693668 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4550366367029 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73513586 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73513586 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58077451 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73503609 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295832624655411 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _step‚626 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296267510747136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53257473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35693665 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4566442984448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131581058 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17563808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4717880088576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283528074471424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283528271484928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295832727831861 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4566355047424 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300833953731584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128¬*hÄ. ûÓO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131581058 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17563808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4717880088576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283528074471424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283528271484928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295832727831861 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4566355047424 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300833953731584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291282361464832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73503609 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278810391396352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3637.534582862 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300833848755200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296267510747136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4566442984448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296267493707776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291282344425472 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚7749 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53257704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35693668 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4550366367029 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73513586 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73513586 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58077451 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73503609 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295832624655411 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _step‚626 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batUÎ¥L«ch_start/active_bytes.large_pool.allocated‚296267510747136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53257473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35693665 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4566442984448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0òXO2ü- _wandbruntime‚3637  _runtime‚36376±%.Þ&ýØM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58084672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278845302910976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58084672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4550933688671 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35698149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131607553 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58084670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296304621711360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296304604672000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295869767188831 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295869664012269 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131607553 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53264167 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278845278007296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296304621711360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4567012350976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131597560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4566924406784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4718470593536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843716964287994e+09 _runtime‚3637.872861395 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17566250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291318833500160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58084670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296304604672000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 _step‚627 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300871634062336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283563572303872 trainer/global_step‚7749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73512890 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283563773504512 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4718294296576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291318816460800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73512890 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300871634062336Þ`Õã&þÝM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58084672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278845302910976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58084672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4550933688671 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35698149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131607553 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58084670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296304621711360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296304604672000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295869767188831 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295869664012269 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131607553 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53264167 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278845278007296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296304621711360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4567012350976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131597560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4566924406784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4718470593536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843716964287994e+09 _runtime‚3637.872861395 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17566250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291318833500160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58084670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296304604672000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 _step‚627 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300871634062336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283563572303872 trainer/global_step‚7749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73512890 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283563773504512 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4718294296576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291318816460800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73512890 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300871634062336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144óû/ÖY2ÿ- _wandbruntime‚3637  _runtime‚3637Ì?çb€ÿ 2loss/FBt1/assignment_loss‚0.017688232577103603 /loss/FRt2/detection_loss‚1.4457771375903166 $ _timestamp‚1.784371696430624e+09 2loss/SRqqt1/assignment_loss‚0.6127928472818777 _runtime‚3637.874170266 _step‚628 0loss/FRt2/assignment_loss‚0.5735970273906109 2loss/SRqqt2/assignment_loss‚0.5762355661045796 /loss/FRt1/detection_loss‚0.6380166264834914 /loss/FBt1/detection_loss‚0.6328094158201448 $loss/total_loss‚8.44354051745718 trainer/global_step‚7749 /loss/FRt1/assignment_loss‚1.671270207634994 1loss/FBt2/assignment_loss‚0.03895791217804896 1loss/SRqqt1/detection_loss‚0.8490420315885335 1loss/SRqqt2/detection_loss‚0.7329043459364222 /loss/FBt2/detection_loss‚0.6544491668710564 epoch‚9Ô¬€3 „ /loss/FRt1/assignment_loss‚1.671270207634994 1loss/FBt2/assignment_loss‚0.03895791217804896 1loss/SRqqt1/detection_loss‚0.8490420315885335 1loss/SRqqt2/detection_loss‚0.7329043459364222 /loss/FBt2/detection_loss‚0.6544491668710564 epoch‚9 2loss/FBt1/assignment_loss‚0.017688232577103603 /loss/FRt2/detection_loss‚1.4457771375903166 $ _timestamp‚1.784371696430624e+09 2loss/SRqqt1/assignment_loss‚0.6127928472818777 _runtime‚3637.874170266 _step‚628 0loss/FRt2/assignment_loss‚0.5735970273906109 2loss/SRqqt2/assignment_loss‚0.5762355661045796 /loss/FRt1/detection_loss‚0.6380166264834914 /loss/FBt1/detection_loss‚0.6328094158201448 $loss/total_loss‚8.44354051745718 trainer/global_step‚7749ôa‹*‹œ‚‚(:‘ ÷³íÒ”ºŒ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚174.796' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.959199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0دŸ×ヂ(:Ø ÷³íÒ½Þ†Ž network.recv‚ 1.183105e+06 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 81.91796875) proc.memory.percent‚16.44766330718994 proc.cpu.threads‚17* disk./.usagePercent‚39.614909217270686$ memory_percent‚99.20401573181152# proc.memory.rssMB‚ 2694.78515625! disk.nvme0n1.in‚ 4893.66015625% disk.nvme0n1.out‚5257.67529296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0" disk.dm-0.out‚5257.67529296875$ disk./.usageGB‚590.2907752990723 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 cpu‚26.81130670311986# disk.nvme0c0n1.in‚ 4893.66015625' proc.memory.availableMB‚ 130.4140625' disk.nvme0c0n1.out‚5257.67529296875 network.sent‚ 1.0407717e+07 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4893.66015625 disk.sda.out‚ 81.91796875 disk.rbd1.in‚0 disk.rbd1.out‚ 8.83203125 disk.sda3.out‚ 81.91796875 disk.sda.in‚0.109375 disk.sda1.in‚0œã¬2„- _wandbruntime‚3659  _runtime‚3659Ç+©–Ç…Á trainer/global_step‚7799 % _timestamp‚1.7843717182232702e+09 _runtime‚3659.667016448 _step‚629 & lr-AdamW/pg1‚0.0007832611097352255 & lr-AdamW/pg2‚0.0007832611097352255K<0ÂÌ†Æ _step‚629 & lr-AdamW/pg1‚0.0007832611097352255 & lr-AdamW/pg2‚0.0007832611097352255 trainer/global_step‚7799 % _timestamp‚1.7843717182232702e+09 _runtime‚3659.667016448õèszæ2‡- _wandbruntime‚3659  _runtime‚3659<ýÍIˆÎO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298123058958336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58438501 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73979488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298123058958336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4578764346737 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132408010 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58438501 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298123041918976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297684727577969 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4594943209472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚7799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302718002167808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚280555942223872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58438499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302718002167808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17672203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132417989 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 % _timestamp‚1.7843717182242842e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285303063419904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293105963231232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293105946191872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73969511 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35917865 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @:DeviceStaÈà?Ö‹ tsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3659.668249806 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4594855272448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53590068 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73979488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58438499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4594855272448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285303266724864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280555967127552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4594943209472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4747299597312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302717897191424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35917868 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73969511 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35917865 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3659.668249806 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4594855272448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53590068 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73979488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58438499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4594855272448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285303266724864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280555967127552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4594943209472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4747299597312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302717897191424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35917868 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297684727577969 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4594943209472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚7799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302718002167808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚280555942223872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58438499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302718002167808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17672203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132417989 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 % _timestamp‚1.7843717182242842e+09öäPB⛊‚(: †´íÒïò¹Š gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚178.569' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.71379999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c‚ï/Ï‹‚(:Ä †´íÒœ¡¦) disk./.usagePercent‚39.61490230506726 disk.sda2.in‚0 disk.sda3.in‚0.109375# proc.memory.rssMB‚ 2694.78515625 disk.dm-1.in‚0.109375! disk.dm-0.out‚5261.2041015625% proc.memory.availableMB‚ 130.28125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚82.125 disk.drbd2165.out‚0 network.recv‚ 1.187023e+06 disk.rbd0.out‚0 disk.dm-1.out‚82.125$ memory_percent‚99.20482635498047 cpu‚26.808967217933873$ disk.nvme0n1.out‚5261.2041015625 disk.dm-0.in‚ 4893.66015625 disk.sda.out‚82.125 disk.rbd1.out‚ 8.83203125$ disk./.usageGB‚590.2906723022461! disk.nvme0n1.in‚ 4893.66015625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 4893.66015625 network.sent‚ 1.0483211e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.rbd1.in‚0& disk.nvme0c0n1.out‚5261.2041015625) proc.memory.percent‚16.44766330718994 proc.cpu.threads‚17òíWÜ2Œ- _wandbruntime‚3660  _runtime‚3660ú1ÝÞ&ØM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58445720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17674567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17674340 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35922349 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4595512576000 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58445720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132434505 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280590878642176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4595424631808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73978792 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298160169922560 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4579245531177 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298160169922560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298160152883200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58445722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297721766934939 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280590853738496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4747711230976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4595512576000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302755577515008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285338764072960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 % _timestamp‚1.7843717185693274e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73988783 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4579331668379 _runtime‚3660.013356367 _step‚631 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73988783 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53596689 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607EªWñŽÝM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297721766934939 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280590853738496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4747711230976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4595512576000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302755577515008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285338764072960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 % _timestamp‚1.7843717185693274e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73988783 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4579331668379 _runtime‚3660.013356367 _step‚631 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73988783 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53596689 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58445720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17674567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17674340 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35922349 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4595512576000 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58445720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132434505 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280590878642176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4595424631808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73978792 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298160169922560 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4579245531177 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298160169922560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298160152883200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58445722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128÷†šÎ–2- _wandbruntime‚3660  _runtime‚3660Âe+ü 2loss/FBt2/assignment_loss‚0.045622861173793994 0loss/SRqqt2/detection_loss‚0.580871334248597 0loss/FBt1/detection_loss‚0.46604637225114165 /loss/FBt2/detection_loss‚0.6575680383430154 epoch‚9 trainer/global_step‚7799 0loss/FRt2/assignment_loss‚0.6782980966118147 1loss/FBt1/assignment_loss‚0.11085492179238629 /loss/FRt1/detection_loss‚0.6156720454268721 _runtime‚3660.014728227 _step‚632 0loss/FRt1/assignment_loss‚1.6004241049868388 1loss/SRqqt1/detection_loss‚0.9490137941130762 $loss/total_loss‚8.45692897770798 2loss/SRqqt1/assignment_loss‚0.6576227450797602 2loss/SRqqt2/assignment_loss‚0.5545038028407592 /loss/FRt2/detection_loss‚1.5404308608399246 % _timestamp‚1.7843717185708656e+09‘ðD0 ‘† _step‚632 0loss/FRt1/assignment_loss‚1.6004241049868388 1loss/SRqqt1/detection_loss‚0.9490137941130762 $loss/total_loss‚8.45692897770798 2loss/SRqqt1/assignment_loss‚0.6576227450797602 2loss/SRqqt2/assignment_loss‚0.5545038028407592 /loss/FRt2/detection_loss‚1.5404308608399246 % _timestamp‚1.7843717185708656e+09 2loss/FBt2/assignment_loss‚0.045622861173793994 0loss/SRqqt2/detection_loss‚0.580871334248597 0loss/FBt1/detection_loss‚0.46604637225114165 /loss/FBt2/detection_loss‚0.6575680383430154 epoch‚9 trainer/global_step‚7799 0loss/FRt2/assignment_loss‚0.6782980966118147 1loss/FBt1/assignment_loss‚0.11085492179238629 /loss/FRt1/detection_loss‚0.6156720454268721 _runtime‚3660.014728227øõ%͸‘’‚(:† •´íÒ¨³‹ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚171.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨¹[iä“‚(:Ù •´íÒ™®ð network.sent‚ 1.0521385e+07) disk./.usagePercent‚39.61489872096178 disk.sda1.in‚0$ disk./.usageGB‚590.2906188964844 disk.sda3.in‚0.109375 disk.sda.out‚ 82.33984375 disk.dm-1.out‚ 82.33984375" disk.dm-0.out‚5266.61962890625 network.recv‚ 1.189993e+06 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 129.95703125 cpu‚26.806626677544166 disk.sda.in‚0.109375# disk.nvme0c0n1.in‚ 4893.66015625$ memory_percent‚99.20680522918701% disk.nvme0n1.out‚5266.61962890625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 4893.66015625# proc.memory.rssMB‚ 2694.78515625 disk.sda3.out‚ 82.33984375 disk.rbd1.out‚ 8.89453125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0' disk.nvme0c0n1.out‚5266.61962890625 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 4893.66015625) proc.memory.percent‚16.44766330718994v#º2”- _wandbruntime‚3681  _runtime‚3681x€åÅÇ•Á & lr-AdamW/pg1‚0.0007823566002554901 & lr-AdamW/pg2‚0.0007823566002554901 trainer/global_step‚7849 % _timestamp‚1.7843717400841277e+09 _runtime‚3681.527895008 _step‚633j¬§6Ì–Æ & lr-AdamW/pg1‚0.0007823566002554901 & lr-AdamW/pg2‚0.0007823566002554901 trainer/global_step‚7849 % _timestamp‚1.7843717400841277e+09 _runtime‚3681.527895008 _step‚633ù ºð2—- _wandbruntime‚3681  _runtime‚3681ݶgÒ'˜ÌO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299978607169536 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287078345779712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4623355497472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4623355497472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58799551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58799551 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294929564997632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 $ _timestamp‚1.784371740085109e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299978590130176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304602050604032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36142068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4607162326445 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133244941 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58799549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚634 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53923499 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74435413 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282301542858752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74445390 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299536727324077 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4776802920960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133234962 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294929547958272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4623443434496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304601945627648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304601945627648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74435413 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚7849 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299978590130176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17781200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74445390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58799549 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287078144571904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4607076189355 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36142065 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680,r×'™ÑO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133234962 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294929547958272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4623443434496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304601945627648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304601945627648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74435413 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚7849 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299978590130176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17781200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74445390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58799549 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287078144571904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4607076189355 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36142065 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299978607169536 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287078345779712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4623355497472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4623355497472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58799551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58799551 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294929564997632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 $ _timestamp‚1.784371740085109e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299978590130176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304602050604032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36142068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4607162326445 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133244941 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58799549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚634 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53923499 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74435413 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282301542858752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74445390 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299536727324077 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4776802920960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146ú‰õöŽ2š- _wandbruntime‚3681  _runtime‚3681¡ Ð ›ØM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74454685 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300015718133760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58806772 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17783564 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4777219981824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304639730934784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300015701094400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4607729648087 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4777387890176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 _runtime‚3681.867670593 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133261457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287113842263552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299573663504485 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133261457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282336429469696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74454685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287113649451520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4623924856832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53929890 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299573766681047 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36146552 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58806770 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74444694 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300015718133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133251464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚7849 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4624012801024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58806772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.78437174~k¾ 04236012e+09 _step‚635 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17783341 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4624012801024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53930116 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294966019993600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36146549 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133261457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282336429469696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74454685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287113649451520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4623924856832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53929890 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299573766681047 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36146552 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58806770 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74444694 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300015718133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133251464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚7849 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4624012801024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58806772 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843717404236012e+09 _step‚635 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17783341 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4624012801024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53930116 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294966019993600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36146549 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17890157 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301388727070185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4635560306153 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296753149724672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _runtime‚3703.493867962 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288853402191872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301388623893735 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301834138341376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4651855722496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4635474169063 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17890406 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚638 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4651855722496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288853196789760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306485994063872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54256674 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301834155380736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4806103103488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4651943659520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74911292 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74911292 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4651943659520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134071893 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59160599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74901315 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284047093686272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚301834138341376 trainer/global_step‚7899 $ _timestamp‚1.784371762049862e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36366265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284047118589952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296753166764032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306485994063872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306486099040256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54256422 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59160601 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74901315 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59160599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301834155380736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36366268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4806283601920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252šqÊ#…§ÒO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17890157 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301388727070185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4635560306153 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296753149724672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _runtime‚3703.493867962 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288853402191872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301388623893735 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301834138341376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4651855722496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4635474169063 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17890406 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚638 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4651855722496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288853196789760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306485994063872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54256674 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301834155380736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4806103103488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4651943659520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74911292 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74911292 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4651943659520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134071893 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59160599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74901315 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284047093686272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytesYÙÅ„S.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚301834138341376 trainer/global_step‚7899 $ _timestamp‚1.784371762049862e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36366265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284047118589952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296753166764032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306485994063872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306486099040256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54256422 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59160601 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74901315 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59160599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301834155380736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36366268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4806283601920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0þU‡æ2¨- _wandbruntime‚3703  _runtime‚3703ˆ¾Ù;Þ&©ØM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4652425081856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301425663250593 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59167822 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301871249305600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54263067 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚3703.833100084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4806697889792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301871266344960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134088409 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134078416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301871249305600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296789621760000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4636041490593 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284082005200896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚639 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134078416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59167820 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54263299 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134088409 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4652513026048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301871249305600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296789621760000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4636041490593 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284082005200896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚639 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134078416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59167820 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54263299 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134088409 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4652513026048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54263067 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚3703.833100084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4806697889792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301871266344960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134088409 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134078416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308370042500096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4663958285861 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚285792694321152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843717838948572e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308370147476480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75377194 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4680355947520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚285792669417472 =6DeviceStatsMonFAD£itor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59521651 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59521649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4680443884544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75367217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134898845 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75377194 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4680355947520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚285792669417472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59521651 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59521649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4680443884544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75367217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134898845 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308370042500096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4663958285861 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327‚cÒŠÅ2¸- _wandbruntime‚3725  _runtime‚3725”ÅdŸÞ&¹ØM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285827605835776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303277766173263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚303726797516800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285827580932096 % _timestamp‚1.7843717842349627e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54595751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134915361 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75376498 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290663721404416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298613223526400 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 trainer/global_step‚7949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75376498 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4680925306880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4836320963584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4681013251072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚643 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59528870 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36594952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36594949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18001031 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75386489 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298613240565760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚308407827807232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4664439470301 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59528872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75386489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290663926799360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54595983 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59528870 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303726814556160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0P¨QºÝM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134905368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4681013251072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303277662996701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54595751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134915361 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75376498 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290663721404416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298613223526400 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 trainer/global_step‚7949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75376498 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4680925306880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4836320963584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4681013251072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚643 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59528870 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36594952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36594949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18001031 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75386489 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298613240565760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚308407827807232 QBDeviceStatsMonitor.on_train_batch_end/res˜SÉH’erved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4664439470301 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59528872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75386489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290663926799360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54595983 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59528870 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303726814556160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285827605835776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303277766173263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚303726797516800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285827580932096 % _timestamp‚1.7843717842349627e+09 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4692270128479 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59882699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚310254090936320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135725797 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4865025717760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚310254195912704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36814668 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54922280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59882699 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305545234763776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679HÒÕqÙ'ÅÓO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚310254195912704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36814668 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54922280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59882699 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305545234763776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4692270128479 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59882699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚310254090936320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135725797 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4865025717760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0†„ý%2Æ- _wandbruntime‚3747  _runtime‚3747+ÆÃ[ŠÇ×M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚310291876243456 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59889920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135732320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305582362767360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4692837450009 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4709513476096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 % _timestamp‚1.7843718062861996e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135742313 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36819152 QCDeviceStatsMonitor.on_train_batch_end/inacSV8”Stive_split_bytes.all.peak‚ 779415040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4692923587211 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59889922 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4709425531904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4709513476096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59889922 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292438967128576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287573181566976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75842400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54928918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54928691 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚310291771259904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59889920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 trainer/global_step‚7999 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4865615556096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305129765919371 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚310291876243456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305582362767360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287573156663296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4865785561600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135742313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300436842332160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _runtime‚ 3747.73025363 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75842400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305582345728000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305129662742809 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327}ÞM¹â&ÈÜM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135732320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305582362767360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4692837450009 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4709513476096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 % _timestamp‚1.7843718062861996e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135742313 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36819152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4692923587211 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59889922 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4709425531904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4709513476096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59889922 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292438967128576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287573181566976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75842400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54928918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54928691 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚310291771259904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59889920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 trainer/global_step‚7999 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4865615556096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305129765919371 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚310291876243456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305582362767360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287573156663296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4865785561600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135742313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300436842332160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _runtime‚ 3747.73025363 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75842400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305582345728000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305129662742809 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚310291876243456 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59889920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101‡aQ¨³2É- _wandbruntime‚3747  _runtime‚3747g’®Ú ʃ 0loss/FRt1/assignment_loss‚1.7536251894436932 2loss/FBt1/assignment_loss‚0.031433130078799505 2loss/SRqqt1/assignment_loss‚0.6327178530310471 /loss/FRt1/detection_loss‚0.5407867078375073 1loss/SRqqt2/detection_loss‚0.6011977442525722 /loss/FBt2/detection_loss‚0.4412116952963285 trainer/global_step‚7999 2loss/SRqqt2/assignment_loss‚0.5484955445089018 2loss/FBt2/assignment_loss‚0.002910685108491777 1loss/SRqqt1/detection_loss‚0.8852578390729972 %loss/total_loss‚8.596943866594014 % _timestamp‚1.7843718062881675e+09 _step‚648 0loss/FRt2/assignment_loss‚0.7547124341130257 /loss/FRt2/detection_loss‚1.6057961609307676 /loss/FBt1/detection_loss‚0.7987988829198811 epoch‚9 _runtime‚3747.731710501Žöþˈ 2loss/FBt1/assignment_loss‚0.031433130078799505 2loss/SRqqt1/assignment_loss‚0.6327178530310471 /loss/FRt1/detection_loss‚0.5407867078375073 1loss/SRqqt2/detection_loss‚0.6011977442525722 /loss/FBt2/detection_loss‚0.4412116952963285 trainer/global_step‚7999 2loss/SRqqt2/assignment_loss‚0.5484955445089018 2loss/FBt2/assignment_loss‚0.002910685108491777 1loss/SRqqt1/detection_loss‚0.8852578390729972 %loss/total_loss‚8.596943866594014 % _timestamp‚1.7843718062881675e+09 _step‚648 0loss/FRt2/assignment_loss‚0.7547124341130257 /loss/FRt2/detection_loss‚1.6057961609307676 /loss/FBt1/detection_loss‚0.7987988829198811 epoch‚9 _runtime‚3747.731710501 0loss/FRt1/assignment_loss‚1.7536251894436932ˆ®Å¯‘Ì‚(:† à´íÒꜢŠ gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UX‚ÓÍ‚(:È à´íÒÐÆ¡Ž disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.rbd1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 4989.26953125 disk.sda3.out‚83.5625 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚5287.642578125 cpu‚26.795570086666526$ disk./.usageGB‚590.2902870178223 disk.sda1.in‚0 disk.sda3.in‚0.109375! disk.nvme0n1.in‚ 4989.26953125$ memory_percent‚99.21205043792725 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2694.78515625 network.sent‚ 1.0766866e+07 network.recv‚ 1.210136e+06 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4989.26953125* disk./.usagePercent‚39.614876448306305 disk.sda1.out‚0 disk.dm-1.in‚0.109375# disk.nvme0n1.out‚5287.642578125( proc.memory.availableMB‚ 129.09765625 disk.dm-1.out‚83.5625 disk.sda.out‚83.5625) proc.memory.percent‚16.44766330718994 disk.dm-0.out‚5287.642578125 disk.rbd1.out‚ 9.01953125äèqœÎ‚(:‘ ï´íÒÅÏ£Š gpu.0.gpu‚51.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 õ#kØÏ‚(:Í ï´íÒÊðî disk.rbd0.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 4989.26953125 disk.dm-0.in‚ 4989.26953125 disk.drbd2165.out‚0$ memory_percent‚99.21994209289551* proc.memory.percent‚16.455602645874023 disk.sda.in‚0.109375" proc.memory.rssMB‚ 2696.0859375 network.sent‚ 1.082958e+07# disk.nvme0c0n1.in‚ 4989.26953125% disk.nvme0c0n1.out‚5290.626953125 disk.sda3.in‚0.109375 proc.cpu.threads‚17 disk.dm-0.out‚5290.626953125 disk.sda.out‚ 83.7578125 disk.rbd1.out‚ 9.08203125 disk.sda3.out‚ 83.7578125 network.recv‚ 1.21448e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 83.7578125 disk.sda1.out‚0) disk./.usagePercent‚39.61487337621589$ disk./.usageGB‚590.2902412414551 disk.drbd2165.in‚0 cpu‚26.806865172642677# disk.nvme0n1.out‚5290.626953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375' proc.memory.availableMB‚ 127.8046875Ñ&Û82Ð- _wandbruntime‚3771  _runtime‚3771a¼Â+3Ñ‚| <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11183353513479233 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1409693956375122 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4437139630317688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12328002601861954 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6185058951377869 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5656000375747681 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06456529349088669 *jet/accuracy_1_of_3‚0.9110284447669983 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13213202357292175 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4682435989379883 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7690098881721497 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984780550003052 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.9035337567329407 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 *jet/accuracy_2_of_3‚0.5060387253761292 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6524134874343872 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10697869956493378 /particle/accuracy_3_of_4‚0.9970238208770752 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 _step‚649 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991559624671936 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.014682539738714695 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1778971403837204 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6572612524032593 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6174919009208679 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913130402565002 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013868214562535286 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12414499372243881 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6199607849121094 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 )jet/accuracy_1_of_4‚0.949469268321991 +particle/sensitivity‚0.6331712007522583 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11642926186323166 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3598247766494751 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6806870102882385 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7244771122932434 *jet/accuracy_2_of_4‚0.7213361263275146 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959271550178528 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.596581220626831 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14884595572948456 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06456529349088669 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6555725932121277 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6519118547439575 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18115389347076416 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024693617597222328 trainer/global_step‚8009 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46590909361839294 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6158027052879333 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7722221612930298 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6209647059440613 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 +jet/accuracy_3_of_4‚0.09668248891830444 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8799999952316284 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 epoch‚9 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.676346480846405 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.636053204536438 / Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.53515625 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42901232838630676 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6468833088874817 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7234196066856384 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 _runtime‚3771.176343592 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26004940271377563 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885833263397217 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7479107975959778 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4350000023841858 particle/accuracy_2_of_3‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36335474252700806 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 *jet/accuracy_2_of_2‚0.1367293745279312 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5498244762420654 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_1_of_1‚0.28012681007385254 (particle/accuracy‚0.8454785346984863 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6185058951377869 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12928727269172668 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 %particle/f_score‚0.63764488697052 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.553519070148468 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1156485453248024 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8572054505348206 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7430306673049927 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7058960199356079 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7494310140609741 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3950115740299225 +jet/accuracy_3_of_3‚0.07440043240785599 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11090646684169769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47875532507896423 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6319071054458618 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45424115657806396 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7339622378349304 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9457465410232544 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5468064546585083 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.845613956451416 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18854166567325592 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.845613956451416 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6736111044883728 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10123417526483536 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8799999952316284 /particle/accuracy_4_of_4‚0.9485178589820862 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1910100281238556 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6136294603347778 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.033640727400779724 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1008232906460762 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975398182868958 /particle/accuracy_3_of_3‚0.9995039105415344 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6211225390434265 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11759157478809357 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31091010570526123 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8457264304161072 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.697012722492218 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )loss/val_total_loss‚8.172338665670315 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6579844951629639 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947346448898315 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004951915238052607 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7690098881721497 particle/accuracy_1_of_4‚1 $ _timestamp‚1.784371829731878e+09 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021307790651917458 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8452686667442322 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002981151919811964 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4236111342906952 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8869689702987671 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_2‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12714894115924835 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957764148712158 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6933821439743042 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12109953910112381 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7105361223220825 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888193011283875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7435751557350159 8 Purity/2FRt1SRqqt0FBt/FRt_NVê¥Ý purity‚0.14406262338161469 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934656023979187 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12536239624023438 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08863988518714905 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07644614577293396 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07221850007772446 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1156485453248024 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3293052017688751 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21872708201408386 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6761101484298706 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9965643286705017 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7363651990890503 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07137399166822433 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6967330574989319 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1303711235523224 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11067038774490356 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9826262593269348 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6184155941009521 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6944223642349243 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07484644651412964 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9234927296638489 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4060862362384796 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9971488118171692 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8605027794837952 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6275492310523987 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16253186762332916 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8807585835456848 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0x¦¼ >Ò‡| 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11067038774490356 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9826262593269348 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6184155941009521 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6944223642349243 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07484644651412964 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9234927296638489 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4060862362384796 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9971488118171692 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8605027794837952 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6275492310523987 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16253186762332916 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8807585835456848 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11183353513479233 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1409693956375122 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4437139630317688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12328002601861954 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6185058951377869 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5656000375747681 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06456529349088669 *jet/accuracy_1_of_3‚0.9110284447669983 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13213202357292175 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4682435989379883 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7690098881721497 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984780550003052 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.9035337567329407 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 *jet/accuracy_2_of_3‚0.5060387253761292 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6524134874343872 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10697869956493378 /particle/accuracy_3_of_4‚0.9970238208770752 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 _step‚649 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991559624671936 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.014682539738714695 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1778971403837204 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6572612524032593 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6174919009208679 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913130402565002 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013868214562535286 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12414499372243881 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6199607849121094 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 )jet/accuracy_1_of_4‚0.949469268321991 +particle/sensitivity‚0.6331712007522583 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11642926186323166 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3598247766494751 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6806870102882385 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7244771122932434 *jet/accuracy_2_of_4‚0.7213361263275146 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959271550178528 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.596581220626831 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14884595572948456 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06456529349088669 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6555725932121277 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6519118547439575 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18115389347076416 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024693617597222328 trainer/global_step‚8009 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46590909361839294 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6158027052879333 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7722221612930298 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6209647059440613 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 +jet/accuracy_3_of_4‚0.09668248891830444 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8799999952316284 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 epoch‚9 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.676346480846405 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.636053204536438 / Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.53515625 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42901232838630676 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6468833088874817 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7234196066856384 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 _runtime‚3771.176343592 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26004940271377563 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885833263397217 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7479107975959778 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4350000023841858 particle/accuracy_2_of_3‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36335474252700806 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 *jet/accuracy_2_of_2‚0.1367293745279312 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5498244762420654 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_1_of_1‚0.28012681007385254 (particle/accuracy‚0.8454785346984863 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6185058951377869 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12928727269172668 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 %particle/f_score‚0.63764488697052 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.553519070148468 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1156485453248024 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8572054505348206 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7430306673049927 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7058960199356079 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7494310140609741 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3950115740299225 +jet/accuracy_3_of_3‚0.07440043240785599 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11090646684169769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47875532507896423 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6319071054458618 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45424115657806396 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7339622378349304 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9457465410232544 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5468064546585083 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.845613956451416 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18854166567325592 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.845613956451416 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6736111044883728 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10123417526483536 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8799999952316284 /particle/accuracy_4_of_4‚0.9485178589820862 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1910100281238556 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6136294603347778 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.033640727400779724 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1008232906460762 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975398182868958 /particle/accuracy_3_of_3‚0.9995039105415344 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6211225390434265 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11759157478809357 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31091010570526123 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8457264304161072 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.697012722492218 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )loss/val_total_loss‚8.172338665670315 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6579844951629639 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947346448898315 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004951915238052607 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7690098881721497 particle/accuracy_1_of_4‚1 $ _timestamp‚1.784371829731878e+09 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021307790651917458 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8452686667442322 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002981151919811964 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4236111342906952 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8869689702987671 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_2‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12714894115924835 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957764148712158 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6933821439743042 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12109953910112381 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7105361223220825 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888193011283875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7435751557350159 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14406262338161469 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934656023979187 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12536239624023438 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08863988518714905 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07644614577293396 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07221850007772446 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1156485453248024 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3293052017688751 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21872708201408386 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6761101484298706 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9965643286705017 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7363651990890503 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07137399166822433 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6967330574989319 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1303711235523224 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436‰WúTœÓ‚(:‘ þ´íÒҔNj gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚178.328' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.665600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lýmâÔ‚(:× þ´íÒÞ–°Œ$ memory_percent‚99.24230575561523 disk.dm-0.in‚ 5105.76171875# disk.nvme0c0n1.in‚ 5105.76171875% disk.nvme0n1.out‚5296.23681640625$ disk./.usageGB‚590.2905120849609 disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 5105.76171875) disk./.usagePercent‚39.61489155275083" disk.dm-0.out‚5296.23681640625 cpu‚26.81145584995706 network.sent‚ 1.0834418e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 83.97265625 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.out‚ 9.08203125' disk.nvme0c0n1.out‚5296.23681640625 disk.sda3.out‚ 83.97265625 disk.sda.out‚ 83.97265625# proc.memory.rssMB‚ 2700.05078125* proc.memory.percent‚16.479802131652832 proc.cpu.threads‚17 network.recv‚ 1.217802e+06 disk.rbd0.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 124.140625üy­2Õ- _wandbruntime‚3788  _runtime‚3788¿?ÓÇÖÁ % _timestamp‚1.7843718470817125e+09 _runtime‚3788.526693341 _step‚650 & lr-AdamW/pg1‚0.0007786813153689545 & lr-AdamW/pg2‚0.0007786813153689545 trainer/global_step‚8049q‘øÛÌ×Æ & lr-AdamW/pg2‚0.0007786813153689545 trainer/global_step‚8049 % _timestamp‚1.7843718470817125e+09 _runtime‚3788.526693341 _step‚650 & lr-AdamW/pg1‚0.0007786813153689545Š~ÑÓ2Ø- _wandbruntime‚3788  _runtime‚3788‚Ï®.Ô'ÙÎO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55430207 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 % _timestamp‚1.7843718470824559e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4753958189568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308361779367936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308361762328576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76442594 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313115632581120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60478279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18286759 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290198334304256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76442594 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60478277 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76452571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76452571 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4911211429888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37143445 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4737242846718 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60478279 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚3788.526808709 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313115737557504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4737156709628 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024û^‡ÚÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55430207 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 % _timestamp‚1.7843718470824559e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4753958189568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308361779367936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308361762328576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76442594 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313115632581120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60478279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18286759 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290198334304256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76442594 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60478277 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76452571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76452571 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4911211429888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37143445 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4737242846718 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60478279 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _runtime‚3788.526808709 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313115737557504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4737156709628 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128‹ úH™2Û- _wandbruntime‚3788  _runtime‚3788ñ3YÞÞ&ÜØM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚8049 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18288769 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313153417888256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136937373 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307955368964278 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4911620470784 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76451875 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4754527556096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60485498 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4754527556096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308398890332160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136947366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290233245818880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307955472140840 _runtime‚3788.865206429 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60485498 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18288993 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55436698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76451875 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4911803059200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136947366 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37147932 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308398890332160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _step‚652 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4754527556096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308398890332160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136947366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290233245818880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307955472140840 _runtime‚3788.865206429 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60485498 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18288993 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55436698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76451875 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4911803059200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136947366 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37147932 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308398890332160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _step‚652 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚8049 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18288769 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313153417888256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136937373 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307955368964278 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4911620470784 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76451875 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4754527556096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60485498 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346Œ–cãF2Þ- _wandbruntime‚3788  _runtime‚3788×g ¼ß /loss/FRt1/assignment_loss‚1.702950873856361 1loss/FBt1/assignment_loss‚0.00587120096901792 /loss/FRt2/detection_loss‚1.4634418057060792 0loss/FBt1/detection_loss‚0.36055513587780297 epoch‚10 % _timestamp‚1.7843718474227216e+09 _step‚653 0loss/FRt2/assignment_loss‚0.7153711472321482 1loss/SRqqt2/detection_loss‚0.6101563897331705 .loss/FBt2/detection_loss‚0.618177360676996 %loss/total_loss‚7.909955860570539 trainer/global_step‚8049 2loss/SRqqt2/assignment_loss‚0.5222328246411535 _runtime‚3788.866259447 2loss/SRqqt1/assignment_loss‚0.5330280903130037 1loss/FBt2/assignment_loss‚0.03937743884828174 /loss/FRt1/detection_loss‚0.5637008108477088 1loss/SRqqt1/detection_loss‚0.7750927818688164ÏŠ!m à† %loss/total_loss‚7.909955860570539 trainer/global_step‚8049 2loss/SRqqt2/assignment_loss‚0.5222328246411535 _runtime‚3788.866259447 2loss/SRqqt1/assignment_loss‚0.5330280903130037 1loss/FBt2/assignment_loss‚0.03937743884828174 /loss/FRt1/detection_loss‚0.5637008108477088 1loss/SRqqt1/detection_loss‚0.7750927818688164 /loss/FRt1/assignment_loss‚1.702950873856361 1loss/FBt1/assignment_loss‚0.00587120096901792 /loss/FRt2/detection_loss‚1.4634418057060792 0loss/FBt1/detection_loss‚0.36055513587780297 epoch‚10 % _timestamp‚1.7843718474227216e+09 _step‚653 0loss/FRt2/assignment_loss‚0.7153711472321482 1loss/SRqqt2/detection_loss‚0.6101563897331705 .loss/FBt2/detection_loss‚0.618177360676996Kë¡$á‚(:„ µíÒ’žØ‹ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚186.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÔV§Ýâ‚(:Ò µíÒ–ûìŒ network.sent‚ 1.089221e+07 disk.dm-0.out‚5299.857421875 disk.rbd1.out‚ 9.14453125# disk.nvme0c0n1.in‚ 5105.76171875 disk.rbd0.in‚0* disk./.usagePercent‚39.614811422392606 disk.dm-0.in‚ 5105.76171875 cpu‚26.80918035777464 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 123.7265625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0.109375 disk.rbd1.in‚0! disk.nvme0n1.in‚ 5105.76171875# disk.nvme0n1.out‚5299.857421875 proc.cpu.threads‚17 disk.sda.out‚ 84.21484375 disk.drbd2165.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2700.05078125 disk.sda3.out‚ 84.21484375% disk.nvme0c0n1.out‚5299.857421875$ memory_percent‚99.24483299255371* proc.memory.percent‚16.479802131652832 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375 network.recv‚ 1.222115e+06$ disk./.usageGB‚590.2893180847168 disk.dm-1.out‚ 84.21484375׶Ñã‚(:… œµíÒ‘¡ºŠ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚170.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.089 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0flÖä‚(:Ë œµíÒø³§ network.recv‚ 1.226469e+06 disk.dm-1.out‚84.6875 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.sda3.out‚84.6875 disk.sda.in‚0.109375 disk.rbd1.in‚0# memory_percent‚99.2457628250122* disk./.usagePercent‚39.614824222769315 disk.rbd0.out‚0* proc.memory.percent‚16.479802131652832 disk.sda.out‚84.6875& disk.nvme0c0n1.out‚5306.1552734375 disk.rbd0.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 5105.76171875 proc.cpu.threads‚17! disk.dm-0.out‚5306.1552734375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 123.57421875# proc.memory.rssMB‚ 2700.05078125$ disk./.usageGB‚590.2895088195801 disk.dm-0.in‚ 5105.76171875 disk.rbd1.out‚ 9.14453125$ disk.nvme0n1.out‚5306.1552734375 network.sent‚ 1.0954953e+07# disk.nvme0c0n1.in‚ 5105.76171875 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚26.807089199255138f+ð2å- _wandbruntime‚3810  _runtime‚3810Š fæ¿ & lr-AdamW/pg1‚êÄ®0.0007777482559894744 & lr-AdamW/pg2‚0.0007777482559894744 trainer/global_step‚8099 $ _timestamp‚1.784371868866629e+09 _runtime‚ 3810.31044678 _step‚654éÊçÄ $ _timestamp‚1.784371868866629e+09 _runtime‚ 3810.31044678 _step‚654 & lr-AdamW/pg1‚0.0007777482559894744 & lr-AdamW/pg2‚0.0007777482559894744 trainer/global_step‚8099ŽäßLŽ2è- _wandbruntime‚3810  _runtime‚3810Ù#ïÔ'éÎO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291943934939136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843718688677423e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310217310539776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60839329 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37367648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76918473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76918473 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305004774664192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4782458414592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314999681017344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305004791703552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310217327579136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309770432529978 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4765554689336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314999681017344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137757802 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296884588560896 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _runtime‚3810.314593768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60839327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310217327579136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4782370477568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37367645 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101îÿutÙ'êÓO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310217327579136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309770432529978 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4765554689336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314999681017344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137757802 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296884588560896 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _runtime‚3810.314593768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60839327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310217327579136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4782370477568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37367645 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291943934939136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843718688677423e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310217310539776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60839329 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37367648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76918473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76918473 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305004774664192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4782458414592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314999681017344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305004791703552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146¨ÕÄ2ë- _wandbruntime‚3810  _runtime‚3810ŽŽ×Þ&ìØM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4782939836928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60846548 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137764325 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4766208148068 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305041263738880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8099 _runtime‚3810.653948581 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291978821550080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309807471886948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310254438543360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4783027781120 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137774318 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310254421504000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315037361340928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309807368710386 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782939836928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305041246699520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315037466324480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137764325 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60846548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55769848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137774318 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37372132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296919888620032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296920089820672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4941067069952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4766122010866 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315037466324480 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60846550 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76917777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60846550 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76927768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146w’çUaíÝM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137774318 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310254421504000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315037361340928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309807368710386 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782939836928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305041246699520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315037466324480 QCDeviceStatsMonitor.on_train_P_ ‚batch_end/inactive_split_bytes.all.peak‚ 779415040 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137764325 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60846548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55769848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137774318 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37372132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296919888620032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296920089820672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4941067069952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4766122010866 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315037466324480 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60846550 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76917777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60846550 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76927768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4782939836928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60846548 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137764325 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4766208148068 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305041263738880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8099 _runtime‚3810.653948581 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291978821550080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309807471886948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310254438543360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4783027781120 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316883729453568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61200379 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306828376430592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293689510670336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4969890058752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37591848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4810870702592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 $ _timestamp‚1.784371891669101e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4810958639616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37591845 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311622329099636 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77384375 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 _runtime‚3833.113112488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61200377 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293689485766656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18504874 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4970066362880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312072875790336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4794038806134 _step‚659 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61200377 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312072875790336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77374398 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298659577033216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138584754 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 trainer/global_step‚8149 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77384375 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4810870702592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4810958639616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316883729453568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4793952669044 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56096487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306828393469952 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312072858750976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640ËëŒØ'øÒO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312072858750976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316883834429952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293689510670336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4969890058752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37591848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4810870702592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 $ _timestamp‚1.784371891669101e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4810958639616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37591845 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311622329099636 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77384375 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 _runtime‚3833.113112488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61200377 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293689485766656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18504874 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4970066362880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312072875790336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4794038806134 _step‚659 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61200377 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312072875790336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77374398 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298659577033216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138584754 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 trainer/global_step‚8149 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77384375 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4810870702592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4810958639616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316883729453568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4793952669044 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56096487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306828393469952 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312072858750976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316883834429952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77374398 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56096722 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138574775 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316883729453568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61200379 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306828376430592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77393670 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4811528006144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138591277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.f›¨£4]$reed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312109986754560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306864848465920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61207598 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4794519990574 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138591277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18506832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37596332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18507067 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚8149 _runtime‚3833.460163848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 % _timestamp‚1.7843718920161269e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138601270 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138601270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316921514760704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77383679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312109969715200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4794606127776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293724397281280 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77383679 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298694878616064 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61207600 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306864865505280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56103161 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4970655534592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚311659368456494 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316921409777152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¨Ÿæã&ûÝM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77383679 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298694878616064 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61207600 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306864865505280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56103161 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4970655534592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚311659368456494 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316921409777152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316921514760704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311659471633056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77393670 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4811528006144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138591277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312109986754560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306864848465920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61207598 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4794519990574 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138591277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18506832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37596332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18507067 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 trainer/global_step‚8149 _runtime‚3833.460163848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 % _timestamp‚1.7843718920161269e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138601270 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138601270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316921514760704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77383679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312109969715200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4794606127776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293724397281280”™ÊR¦2ü- _wandbruntime‚3833  _runtime‚3833ëìIF ý„ .loss/FRt2/detection_loss‚1.509881412142564 trainer/global_step‚8149 2loss/SRqqt1/assignment_loss‚0.5284219654532487 3loss/FBt2/assignment_loss‚0.0005562130649607866 /loss/FRt1/detection_loss‚0.5569208045028375 /loss/FBt1/detection_loss‚0.5951004375850684 /loss/FBt2/detection_loss‚0.5288332782630973 epoch‚10 % _timestamp‚1.7843718920180953e+09 _step‚661 0loss/FRt2/assignment_loss‚0.7113831020856687 3loss/SRqqt2/assignment_loss‚0.43875134627552104 1loss/SRqqt2/detection_loss‚0.6587365672580505 _runtime‚3833.461640305 1loss/SRqqt1/detection_loss‚0.9039828769983889 #loss/total_loss‚8.1447008007871 0loss/FRt1/assignment_loss‚1.7091923265252262 3loss/FBt1/assignment_loss‚0.0029404706324680624@lF"þ‰ .loss/FRt2/detection_loss‚1.509881412142564 trainer/global_step‚8149 2loss/SRqqt1/assignment_loss‚0.5284219654532487 3loss/FBt2/assignment_loss‚0.0005562130649607866 /loss/FRt1/detection_loss‚0.5569208045028375 /loss/FBt1/detection_loss‚0.5951004375850684 /loss/FBt2/detection_loss‚0.5288332782630973 epoch‚10 % _timestamp‚1.7843718920180953e+09 _step‚661 0loss/FRt2/assignment_loss‚0.7113831020856687 3loss/SRqqt2/assignment_loss‚0.43875134627552104 1loss/SRqqt2/detection_loss‚0.6587365672580505 _runtime‚3833.461640305 1loss/SRqqt1/detection_loss‚0.9039828769983889 #loss/total_loss‚8.1447008007871 0loss/FRt1/assignment_loss‚1.7091923265252262 3loss/FBt1/assignment_loss‚0.0029404706324680624•Dõêÿ‚(:… ºµíÒ省‹ gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚187.975' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª›Ê±Ú€‚(:Ï ºµíÒÁ†úŽ disk.rbd0.out‚0 disk.sda2.in‚0" disk.dm-0.out‚5318.00048828125 disk.sda3.in‚0.109375 proc.cpu.threads‚17$ disk./.usageGB‚590.2917976379395 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.479802131652832 network.sent‚ 1.1035676e+07 disk.nvme0n1.in‚ 5185.5703125 disk.rbd0.in‚0 disk.sda.in‚0.109375' proc.memory.availableMB‚ 124.0859375 disk.sda3.out‚ 85.390625 disk.sda.out‚ 85.390625 disk.rbd1.in‚0' disk.nvme0c0n1.out‚5318.00048828125 disk.sda1.out‚0 disk.dm-1.in‚0.109375) disk./.usagePercent‚39.61497782728986 disk.rbd1.out‚ 9.26953125% disk.nvme0n1.out‚5318.00048828125 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 5185.5703125 disk.dm-1.out‚ 85.390625$ memory_percent‚99.24263954162598 cpu‚26.81130570732265 network.recv‚ 1.233875e+06 disk.dm-0.in‚ 5185.5703125# proc.memory.rssMB‚ 2700.05078125ñˆ¯P‚(:… ɵíÒŠ÷œŠ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚184.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.947 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­­†‚ׂ‚(:Ì ɵíÒ½²ü disk.sda2.out‚0& disk.nvme0c0n1.out‚5324.7490234375 network.sent‚ 1.1085587e+07 network.recv‚ 1.237904e+06$ disk./.usageGB‚590.2897834777832 disk.sda3.out‚ 85.5078125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 5185.578125' proc.memory.availableMB‚ 123.6484375 disk.sda1.out‚0 disk.dm-1.out‚ 85.5078125# proc.memory.rssMB‚ 2700.05078125 disk.sda1.in‚0! disk.dm-0.out‚5324.7490234375 disk.sda.out‚ 85.5078125 disk.rbd0.out‚0$ memory_percent‚99.24530982971191* proc.memory.percent‚16.479802131652832 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 5185.578125 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.rbd1.out‚ 9.33203125 disk.rbd0.in‚0) disk./.usagePercent‚39.61484265531178 disk.sda2.in‚0$ disk.nvme0n1.out‚5324.7490234375 cpu‚26.81340099603777 proc.cpu.threads‚17 disk.dm-0.in‚ 5185.578125 disk.sda3.in‚0.109375˜Î+2ƒ- _wandbruntime‚3855  _runtime‚3855K¦pÅ„¿ % lr-AdamW/pg1‚0.000775865155735866 % lr-AdamW/pg2‚0.000775865155735866 trainer/global_step‚8199 % _timestamp‚1.7843719141786485e+09 _runtime‚3855.622504201 _step‚662Ù=ÝÊ…Ä trainer/global_step‚8199 % _timestamp‚1.7843719141786485e+09 _runtime‚3855.622504201 _step‚662 % lr-AdamW/pg1‚0.000775865155735866 % lr-AdamW/pg2‚0.000775865155735866–šeô2†- _wandbruntime‚3855  _runtime‚3855ð²B¤O!‡ÎO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318767882866176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313928406962176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37816048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61561429 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56430359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999533913088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313474328845744 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61561427 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313928424001536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313928406962176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4839370927616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77840300 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56430125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4822436785842 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318767777889792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18614080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308651978196992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4822350648752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61561427 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77840300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999359706112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318767882866176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308651995236352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139401727 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4839458864640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4839458864640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37816045 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77850277 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStÏêÃý…atsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318767777889792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18614311 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300434421203968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61561429 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313928424001536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839370927616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295435086401536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139411706 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 % _timestamp‚1.7843719141795948e+091‚šÐÙ'ˆÓO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318767882866176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313928406962176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37816048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61561429 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56430359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999533913088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313474328845744 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61561427 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313928424001536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313928406962176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4839370927616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77840300 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56430125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4822436785842 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318767777889792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18614080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308651978196992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4822350648752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61561427 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77840300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999359706112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318767882866176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308651995236352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139401727 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4839458864640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4839458864640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37816045 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77850277 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318767777889792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18614311 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300434421203968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61561429 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313928424001536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839370927616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295435086401536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139411706 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 % _timestamp‚1.7843719141795948e+09—á6èO2‰- _wandbruntime‚3855  _runtime‚3855N)ÒÞ&ŠØM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77859572 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313965534965760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139428222 _runtime‚3855.970579163 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56436777 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4839940286976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308688450232320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61568650 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4823004107484 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77849581 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77849581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37820529 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4999952624640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318805458213376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300470120546304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139418229 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313965517926400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚8199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37820532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300469925637120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313511471379164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843719145264409e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139418229 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313511368202602 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295469997916160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139428222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61568648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318805563196928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313965517926400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61568648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4822917970282 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ö.Õ­ã&‹ÝM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139428222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61568648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318805563196928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313965517926400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61568648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4822917970282 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77859572 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313965534965760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139428222 _runtime‚3855.970579163 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56436777 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4839940286976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308688450232320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61568650 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4823004107484 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77849581 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77849581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37820529 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4999952624640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318805458213376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300470120546304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139418229 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313965517926400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚8199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37820532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300469925637120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313511471379164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843719145264409e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139418229 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313511368202602 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295469997916160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504˜}¯C2Œ- _wandbruntime‚3855  _runtime‚3855þkÏ † 0loss/FRt2/assignment_loss‚0.8879948103983786 3loss/FBt1/assignment_loss‚6.204625497474581e-06 %loss/total_loss‚8.900021100577662 epoch‚10 2loss/SRqqt2/assignment_loss‚0.5407190876646991 0loss/FBt2/detection_loss‚0.45126215672035785 trainer/global_step‚8199 0loss/FRt1/assignment_loss‚1.7580757344936928 2loss/FBt2/assignment_loss‚0.002705221241780526 /loss/FRt1/detection_loss‚0.5465364034848244 /loss/FBt1/detection_loss‚0.5792870247736573 _runtime‚3855.971727922 2loss/SRqqt1/assignment_loss‚0.7733736729043462 /loss/FRt2/detection_loss‚1.6211223721629675 1loss/SRqqt1/detection_loss‚1.0512753691067283 1loss/SRqqt2/detection_loss‚0.6876630430007323 % _timestamp‚1.7843719145281718e+09 _step‚665Ï*9Ž‹ 2loss/SRqqt1/assignment_loss‚0.773373672904346’Žw"Ø2 /loss/FRt2/detection_loss‚1.6211223721629675 1loss/SRqqt1/detection_loss‚1.0512753691067283 1loss/SRqqt2/detection_loss‚0.6876630430007323 % _timestamp‚1.7843719145281718e+09 _step‚665 0loss/FRt2/assignment_loss‚0.8879948103983786 3loss/FBt1/assignment_loss‚6.204625497474581e-06 %loss/total_loss‚8.900021100577662 epoch‚10 2loss/SRqqt2/assignment_loss‚0.5407190876646991 0loss/FBt2/detection_loss‚0.45126215672035785 trainer/global_step‚8199 0loss/FRt1/assignment_loss‚1.7580757344936928 2loss/FBt2/assignment_loss‚0.002705221241780526 /loss/FRt1/detection_loss‚0.5465364034848244 /loss/FBt1/detection_loss‚0.5792870247736573 _runtime‚3855.971727922™ÆW§ß‘‚(:† صíÒøڊ gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚172.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š‘nÒ‚(:Ç صíÒ¥¡Œ network.recv‚ 1.241133e+06# disk./.usageGB‚590.290096282959 disk.sda3.in‚0.109375 disk.sda.in‚0.109375! disk.nvme0c0n1.in‚ 5185.578125 disk.dm-1.out‚ 86.046875 disk.drbd2165.out‚0 disk.sda1.out‚0# proc.memory.availableMB‚123.625 disk.dm-0.in‚ 5185.578125' disk.nvme0c0n1.out‚5328.49951171875 disk.nvme0n1.in‚ 5185.578125 disk.rbd0.out‚0" disk.dm-0.out‚5328.49951171875 disk.sda3.out‚ 86.046875$ memory_percent‚99.24545288085938 network.sent‚ 1.1090431e+07 disk.sda.out‚ 86.046875 disk.rbd1.in‚0% disk.nvme0n1.out‚5328.49951171875* proc.memory.percent‚16.479802131652832 disk.rbd0.in‚0 disk.dm-1.in‚0.109375# proc.memory.rssMB‚ 2700.05078125 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚26.8159395170845 proc.cpu.threads‚17* disk./.usagePercent‚39.614863647929596 disk.drbd2165.in‚0 disk.rbd1.out‚ 9.33203125 disk.sda2.out‚0mž¢h2‘- _wandbruntime‚3878  _runtime‚3878Þ1`Ç’Á _step‚666 & lr-AdamW/pg1‚0.0007749151450234108 & lr-AdamW/pg2‚0.0007749151450234108 trainer/global_step‚8249 % _timestamp‚1.7843719369626572e+09 _runtime‚3878.406542082c}Ì“Æ % _timestamp‚1.7843719369626572e+09 _runtime‚3878.406542082 _step‚666 & lr-AdamW/pg1‚0.0007749151450234108 & lr-AdamW/pg2‚0.0007749151450234108 trainer/global_step‚8249šñ ·2”- _wandbruntime‚3878  _runtime‚3878ÚØîòÔ'•ÎO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4850748628460 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4867871152640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140238658 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302209612508160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297180637229056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚315783955173376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5028780362752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320651826326016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18722992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5028950375424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4867871152640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚3878.407807851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320651931302400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18722774 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61922477 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61922479 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 % _timestamp‚1.7843719369637883e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61922477 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚315783955173376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297180662132736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4850834765550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4867959089664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61922479 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚315783972212736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56763019 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38040248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38040245 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310475597002752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320651826326016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78306202 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78316179 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315326431768302 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315326328591852 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚315783972212736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78306202 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚667j™xèÙ'–ÓO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5028950375424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4867871152640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚3878.407807851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320651931302400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18722774 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61922477 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61922479 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 % _timestamp‚1.7843719369637883e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61922477 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚315783955173376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297180662132736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4850834765550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4867959089664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61922479 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚315783972212736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56763019 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38040248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38040245 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310475597002752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320651826326016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78306202 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78316179 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315326431768302 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315326328591852 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚315783972212736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78306202 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚667 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4850748628460 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4867871152640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140238658 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚221 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302209612508160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297180637229056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚315783955173376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5028780362752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320651826326016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18722992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0›…pý2—- _wandbruntime‚3878  _runtime‚3878Éä×­¢%˜ØM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320689611633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 _runtime‚3878.751199214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56769631 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310512051998720 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18724902 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78315483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38044732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4868528456192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4868440512000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚315821083176960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61929700 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5029369766912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38044729 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78315483 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18725122 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚8249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚315821083176960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5029539772416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315363367948710 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302244918510592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61929700 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4851402087192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297215548743680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140255174 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 % _timestamp‚1.7843719373070848e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302245113419776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140245181 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320689506649600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320689611633152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78325474 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78325474 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297215573647360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚315821066137600 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315363471125272 OBDeviceStatsMonitæžO<or.on_train_batch_end/inactive_split.all.allocated‚56769854 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚668 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192p±öÙã&™ÝM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4868440512000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚315821083176960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61929700 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5029369766912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38044729 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78315483 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18725122 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚8249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚315821083176960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5029539772416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315363367948710 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302244918510592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61929700 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4851402087192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297215548743680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140255174 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 % _timestamp‚1.7843719373070848e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302245113419776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140245181 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320689506649600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320689611633152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78325474 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78325474 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297215573647360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚315821066137600 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315363471125272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56769854 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚668 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320689611633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 _runtime‚3878.751199214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56769631 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310512051998720 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18724902 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78315483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38044732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4868528456192œ„܃2š- _wandbruntime‚3878  _runtime‚3878¦Ð¾» ›„ .loss/FRt2/detection_loss‚1.390936585125719 1loss/SRqqt1/detection_loss‚0.7990635850144849 % _timestamp‚1.7843719373084724e+09 1loss/SRqqt2/detection_loss‚0.6567647322052106 %loss/total_loss‚7.712464705608125 _runtime‚3878.752439476 _step‚669 0loss/FRt2/assignment_loss‚0.5624133558340476 3loss/SRqqt1/assignment_loss‚0.43751386922479096 2loss/SRqqt2/assignment_loss‚0.4445906122114223 2loss/FBt1/assignment_loss‚0.012250659924390561 /loss/FBt1/detection_loss‚0.8122963891609719 /loss/FBt2/detection_loss‚0.3369379917002173 epoch‚10 trainer/global_step‚8249 0loss/FRt1/assignment_loss‚1.6816301777259537 2loss/FBt2/assignment_loss‚0.028616599864731952 /loss/FRt1/detection_loss‚0.5494501476161846©_œ‰ 1loss/SRqqt2/detection_loss‚0.6567647322052106 %loss/total_loss‚7.712464705608125 _runtime‚3878.752439476 _step‚669 0loss/FRt2/assignment_loss‚0.5624133558340476 3loss/SRqqt1/assignment_loss‚0.43751386922479096 2loss/SRqqt2/assignment_loss‚0.4445906122114223 2loss/FBt1/assignment_loss‚0.012250659924390561 /loss/FBt1/detection_loss‚0.8122963891609719 /loss/FBt2/detection_loss‚0.3369379917002173 epoch‚10 trainer/global_step‚8249 0loss/FRt1/assignment_loss‚1.6816301777259537 2loss/FBt2/assignment_loss‚0.028616599864731952 /loss/FRt1/detection_loss‚0.5494501476161846 .loss/FRt2/detection_loss‚1.390936585125719 1loss/SRqqt1/detection_loss‚0.7990635850144849 % _timestamp‚1.7843719373084724e+09”‡Ê‘‚(:† çµíÒôÝæŒ gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚181.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï+Ád´ž‚(:© çµíÒàÖá disk.sda3.out‚86.25 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚ 1.1153362e+07$ disk./.usageGB‚590.2892303466797 disk.rbd1.out‚ 9.39453125 disk.sda1.in‚0 disk.dm-1.in‚0.109375! disk.nvme0c0n1.in‚ 5185.578125 disk.rbd0.in‚0 disk.dm-0.out‚5331.875* proc.memory.percent‚16.479802131652832 disk.nvme0c0n1.out‚5331.875 network.recv‚ 1.2456e+06 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.dm-1.out‚86.25 disk.dm-0.in‚ 5185.578125 cpu‚26.818192419963424 disk.rbd0.out‚0# proc.memory.rssMB‚ 2700.05078125* disk./.usagePercent‚39.614805534219315 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 5185.578125 disk.nvme0n1.out‚5331.875$ memory_percent‚99.24485683441162( proc.memory.availableMB‚ 123.72265625 disk.sda.out‚86.25ꬨœŸ‚(:‘ öµíÒÌÕÅŠ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.643' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.128600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Ç$ÆÏ ‚(:Ä öµíÒèªßŒ# disk.nvme0n1.out‚5338.333984375# proc.memory.rssMB‚ 2700.05078125 disk.sda2.out‚0 disk.sda.out‚ 86.51953125 disk.sda1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.479802131652832 network.recv‚ 1.249932e+06% disk.nvme0c0n1.out‚5338.333984375 network.sent‚ 1.1216151e+07* disk./.usagePercent‚39.649077006820534$ disk./.usageGB‚590.7999000549316 disk.dm-0.out‚5338.333984375 disk.sda3.in‚0.109375 disk.drbd2165.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 123.4765625 disk.sda2.in‚0 disk.nvme0n1.in‚5251.875 cpu‚26.82039360572699 disk.dm-0.in‚5251.875$ memory_percent‚99.24635887145996 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 86.51953125 disk.sda3.out‚ 86.51953125 disk.rbd1.out‚ 9.39453125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.109375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚5251.875C¼_¹2¡- _wandbruntime‚3900  _runtime‚3900Ä®4pÇ¢Á & lr-AdamW/pg2‚0.0007739595141789006 trainer/global_step‚8299 % _timestamp‚1.7843719595286582e+09 _runtime‚3900.972555168 _step‚670 & lr-AdamW/pg1‚0.0007739595141789006ý…šÌ£Æ & lr-AdamW/pg1‚0.0007739595141789006 & lr-AdamW/pg2‚0.0007739595141789006 trainer/global_step‚8299 % _timestamp‚1.7843719595286582e+09 _runtime‚3900.972555168 _step‚670žß%ôf2¤- _wandbruntime‚3900  _runtime‚3900ª™àÕÔ'¥ÎO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62283527 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78782081 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38264448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62283527 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317639520423936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303984657114112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚671 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4896459314688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚322535979738624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298926237863936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4879146608168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317178431514410 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317178328337960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312299198769152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4896459314688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8299 % _timestamp‚1.7843719595296152e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚322535979738624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _runtime‚3900.976603551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78772104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38264445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18832222 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317639520423936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317639503384576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78782081 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141055631 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62283529 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317639503384576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚322535874762240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4879232745258 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146f嶲¦ÓO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚322535874762240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303984458003456 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57096667 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5058245043200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62283529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78772104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5058419250176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4896371377664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141065610 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298926212960256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57096901 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4896371377664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62283527 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317639520423936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303984657114112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚671 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4896459314688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚322535979738624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298926237863936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4879146608168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317178431514410 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317178328337960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312299198769152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4896459314688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8299 % _timestamp‚1.7843719595296152e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚322535979738624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _runtime‚3900.976603551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78772104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38264445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18832222 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317639520423936 \HDeviceStatsMonitor.on_train_batch_start/allocated_byteìŒ ' s.large_pool.freed‚317639503384576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78782081 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141055631 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62283529 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317639503384576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚322535874762240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4879232745258 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62283527 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78782081 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38264448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 J2§- _wandbruntime‚3901  _runtime‚3901`q´Ý&¨×M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _runtime‚3901.319316564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62290748 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317676614348800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4879800066900 _step‚672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78791376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚322573660069376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57103299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317215470871380 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317676614348800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317215367694818 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304020157804544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚322573660069376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18834370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4897028681216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4896940737024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141082126 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62290750 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18834612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298961149378560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298961124474880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4879713929698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141072133 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784371959875268e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78781385 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312335670804480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304019956603904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5059008425984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4897028681216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 48969407370246ãß[â&©ÜM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚322573660069376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18834370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4897028681216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4896940737024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141082126 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62290750 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18834612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298961149378560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298961124474880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4879713929698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141072133 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784371959875268e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78781385 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312335670804480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304019956603904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5059008425984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4897028681216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4896940737024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _runtime‚3901.319316564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62290748 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317676614348800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4879800066900 _step‚672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78791376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚322573660069376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57103299 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317215470871380 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317676614348800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317215367694818 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304020157804544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ù­A@2ª- _wandbruntime‚3901  _runtime‚3901ÞruF« 0loss/FRt2/assignment_loss‚0.6815523649796401 0loss/FBt1/assignment_loss‚0.0061233415332922 1loss/SRqqt1/detection_loss‚0.8378371813905021 /loss/FBt1/detection_loss‚0.3844253620546725 % _timestamp‚1.7843719598768253e+09 /loss/FBt2/detection_loss‚0.4476192793020835 %loss/total_loss‚8.014610956144814 epoch‚10 _step‚673 /loss/FRt1/assignment_loss‚1.704312011122479 2loss/SRqqt2/assignment_loss‚0.5036299755270105 3loss/FBt2/assignment_loss‚0.0026931232960472166 2loss/SRqqt1/assignment_loss‚0.6319742550437185 /loss/FRt1/detection_loss‚0.5595756940310272 .loss/FRt2/detection_loss‚1.533079405458146 1loss/SRqqt2/detection_loss‚0.7217889624061964 trainer/global_step‚8299 _runtime‚3901.320370854Q ž¸ ¬† 2loss/SRqqt2/assignment_loss‚0.5036299755270105 3loss/FBt2/assignment_loss‚0.0026931232960472166 2loss/SRqqt1/assignment_loss‚0.6319742550437185 /loss/FRt1/detection_loss‚0.5595756940310272 .loss/FRt2/detection_loss‚1.533079405458146 1loss/SRqqt2/detection_loss‚0.7217889624061964 trainer/global_step‚8299 _runtime‚3901.320370854 0loss/FRt2/assignment_loss‚0.6815523649796401 0loss/FBt1/assignment_loss‚0.0061233415332922 1loss/SRqqt1/detection_loss‚0.8378371813905021 /loss/FBt1/detection_loss‚0.3844253620546725 % _timestamp‚1.7843719598768253e+09 /loss/FBt2/detection_loss‚0.4476192793020835 %loss/total_loss‚8.014610956144814 epoch‚10 _step‚673 /loss/FRt1/assignment_loss‚1.704312011122479¡ÀË®O‘­‚(:† …¶íÒÇÜÒ‹ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K|º×®‚(:Ì …¶íÒäø* proc.memory.percent‚16.479802131652832 cpu‚26.822213088133534$ disk./.usageGB‚591.0600280761719 disk.sda1.out‚0 network.recv‚ 1.253164e+06* disk./.usagePercent‚39.666534416587915 disk.rbd0.in‚0 disk.sda3.in‚0.109375' disk.nvme0c0n1.out‚5368.36181640625% disk.nvme0n1.out‚5368.36181640625 disk.rbd0.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 86.66015625 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚5251.875 disk.nvme0n1.in‚5251.875 disk.sda2.out‚0 disk.dm-0.in‚5251.875 disk.sda.out‚ 86.66015625 disk.drbd2165.in‚0 disk.rbd1.out‚ 9.45703125" disk.dm-0.out‚5368.36181640625 disk.sda3.out‚ 86.66015625( proc.memory.availableMB‚ 123.46484375 network.sent‚ 1.1221066e+07 disk.sda1.in‚0$ memory_percent‚99.24643039703369 proc.cpu.threads‚17 disk.sda.in‚0.109375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2700.05078125:@Øm2¯- _wandbruntime‚3923  _runtime‚3923¡*Ô Ç°Á & lr-AdamW/pg1‚0.0007729982785087031 & lr-AdamW/pg2‚0.0007729982785087031 trainer/global_step‚8349 % _timestamp‚1.7843719821106873e+09 _runtime‚3923.554622877 _step‚674j]B|Ì±Æ & lr-AdamW/pg1‚0.0007729982785087031 & lr-AdamW/pg2‚0.0007729982785087031 trainer/global_step‚8349 % _timestamp‚1.7843719821106873e+09 _runtime‚3923.554622877 _step‚674¢Åh+‹2²- _wandbruntime‚3923  _runtime‚3923LÜG¹â³ÍO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141882583 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38488648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18942144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324419923198464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79238006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319030431260518 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62644579 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79238006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚3923.555914836 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62644577 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4924871602688 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4924959539712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305759482461184 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324419923198464 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 $ _timestamp‚1.784371982111903e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314122783496192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4924871602688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300671788691456 trainer/global_step‚8349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79247983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319495068635136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319495051595776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38488645 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62644579 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144ÕÕ%ûØ'´ÒO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4924959539712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305759482461184 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324419923198464 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 $ _timestamp‚1.784371982111903e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314122783496192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4924871602688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300671788691456 trainer/global_step‚8349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79247983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319495068635136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319495051595776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38488645 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62644579 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141882583 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38488648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18942144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324419923198464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79238006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319030431260518 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62644579 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79238006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚3923.555914836 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62644577 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4924871602688 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0£>©A‘2µ- _wandbruntime‚3923  _runtime‚3923òêVËÞ&¶ØM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4925528906240 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18944246 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚314159272570880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚3923.899140193 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62651798 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141899085 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843719824550827e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57437378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38493132 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141909078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62651798 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319532179599360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305795179161600 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4925440962048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5088284046336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305794984252416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319067367440926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79257278 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314159255531520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57437151 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4925440962048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18944022 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4925528906240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319532162560000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319532162560000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141909078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79247287 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4908111909406 _step‚676 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62651800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319067470617488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4908198046608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324457603522048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300706725109760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640þ7Í"·ÝM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324457603522048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300706725109760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4925528906240 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18944246 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚314159272570880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚3923.899140193 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62651798 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141899085 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843719824550827e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57437378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38493132 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141909078 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62651798 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319532179599360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305795179161600 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4925440962048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5088284046336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305794984252416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319067367440926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79257278 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314159255531520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57437151 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4925440962048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18944022 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4925528906240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319532162560000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319532162560000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141909078 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceÓ8DrßStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79247287 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4908111909406 _step‚676 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62651800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319067470617488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4908198046608¤¼] ï2¸- _wandbruntime‚3923  _runtime‚3923qáGÈ ¹… /loss/FRt1/detection_loss‚0.5349267234973453 0loss/SRqqt2/detection_loss‚0.687698445964332 0loss/FBt1/detection_loss‚0.47497861128714347 %loss/total_loss‚8.061022184226655 2loss/SRqqt1/assignment_loss‚0.5547392539616268 1loss/FBt2/assignment_loss‚0.03332912171992873 /loss/FRt2/detection_loss‚1.6072851185645185 /loss/FBt2/detection_loss‚0.5301561948361854 trainer/global_step‚8349 0loss/FRt1/assignment_loss‚1.7073576603553084 3loss/SRqqt2/assignment_loss‚0.49159648879469564 3loss/FBt1/assignment_loss‚0.0018453405071498076 % _timestamp‚1.7843719824566808e+09 _runtime‚3923.900228006 _step‚677 0loss/FRt2/assignment_loss‚0.6823876108696212 1loss/SRqqt1/detection_loss‚0.7547216138688004 epoch‚10ãœûºŠ /loss/FRt2/detection_loss‚1.6072851185645185 /loss/FBt2/detection_loss‚0.5301561948361854 trainer/global_step‚8349 0loss/FRt1/assignment_loss‚1.7073576603553084 3loss/SRqqt2/assignment_loss‚0.49159648879469564 3loss/FBt1/assignment_loss‚0.0018453405071498076 % _timestamp‚1.7843719824566808e+09 _runtime‚3923.900228006 _step‚677 0loss/FRt2/assignment_loss‚0.6823876108696212 1loss/SRqqt1/detection_loss‚0.7547216138688004 epoch‚10 /loss/FRt1/detection_loss‚0.5349267234973453 0loss/SRqqt2/detection_loss‚0.687698445964332 0loss/FBt1/detection_loss‚0.47497861128714347 %loss/total_loss‚8.061022184226655 2loss/SRqqt1/assignment_loss‚0.5547392539616268 1loss/FBt2/assignment_loss‚0.03332912171992873¥Α‘»‚(:† ”¶íÒ­ˆê‹ gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚192.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.4578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Õ%Ô¼‚(:É ”¶íÒŸ¾ŒŒ' disk.nvme0c0n1.out‚5372.96337890625 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚26.82448054529265 disk.drbd2165.in‚0" disk.dm-0.out‚5372.96337890625) disk./.usagePercent‚39.66477334075987 disk.sda3.in‚0.109375 disk.sda.out‚ 86.92578125 disk.rbd1.out‚ 9.45703125 disk.nvme0n1.in‚5251.875 disk.sda2.in‚0 network.sent‚ 1.1283995e+07 disk.sda.in‚0.109375 disk.dm-1.in‚0.109375# memory_percent‚99.2448091506958 disk.rbd1.in‚0 disk.nvme0c0n1.in‚5251.875 disk.sda1.in‚0$ disk./.usageGB‚591.0337867736816 disk.sda3.out‚ 86.92578125 disk.drbd2165.out‚0% disk.nvme0n1.out‚5372.96337890625 disk.dm-1.out‚ 86.92578125 disk.dm-0.in‚5251.875( proc.memory.availableMB‚ 123.73046875* proc.memory.percent‚16.479802131652832 network.recv‚ 1.257417e+06 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2700.05078125 proc.cpu.threads‚17`Ú°‘½‚(:† £¶íÒùõЋ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚182.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’vSËƾ‚(:» £¶íÒÚ˜ÓŒ disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17% disk.nvme0c0n1.out‚6166.326171875 network.sent‚ 1.1346621e+07 disk.sda3.out‚ 88.484375 disk.rbd1.in‚0# disk.nvme0n1.out‚6166.326171875& proc.memory.availableMB‚ 123.484375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 9.51953125 network.recv‚ 1.261618e+06 disk.dm-0.out‚6166.326171875* proc.memory.percent‚16.479802131652832( disk./.usagePercent‚39.6149122893611 disk.nvme0c0n1.in‚5251.875 disk.sda2.in‚0 disk.dm-0.in‚5251.875$ memory_percent‚99.24631118774414 cpu‚26.82664573313818 disk.dm-1.out‚ 88.484375$ disk./.usageGB‚590.2908210754395 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.sda.out‚ 88.484375 disk.nvme0n1.in‚5251.875# proc.memory.rssMB‚ 2700.05078125—à2¿- _wandbruntime‚3946  _runtime‚3946jmoÇÀÁ _runtime‚3946.131592154 _step‚678 & lr-AdamW/pg1‚0.0007720314534089588 & lr-AdamW/pg2‚0.0007720314534089588 trainer/global_step‚8399 % _timestamp‚1.7843720046878548e+09<ÂTüÌÁÆ & lr-AdamW/pg2‚0.0007720314534089588 trainer/global_step‚8399 % _timestamp‚1.7843720046878548e+09 _runtime‚3946.131592154 _step‚678 & lr-AdamW/pg1‚0.0007720314534089588¦ä÷q 2Â- _wandbruntime‚3946  _runtime‚3946ê7!Ô'ÃÎO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79703908 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57764698 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321350616846336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚320882431006626 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63005627 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5117338150912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79703908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315946385262592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4936028704674 _step‚679 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326303971634688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321350616846336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307534526269440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4935942567584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326304076611072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142709535 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142719514 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚3946.132611678 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79713885 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321350599806976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚326303971634688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5117161846784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38712845 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315946402301952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4953459764736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63005627 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38712848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4953371827712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321350599806976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4953459764736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79713885 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326304076611072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307534727477248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146îôÖèÙ'ÄÓO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79713885 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321350599806976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚326303971634688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5117161846784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38712845 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315946402301952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4953459764736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63005627 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38712848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4953371827712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321350599806976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4953459764736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79713885 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326304076611072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307534727477248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79703908 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57764698 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321350616846336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚320882431006626 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63005627 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5117338150912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79703908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315946385262592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4936028704674 _step‚679 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326303971634688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321350616846336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307534526269440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4935942567584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326304076611072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142709535 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142719514 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚3946.132611678§62Å- _wandbruntime‚3946  _runtime‚3946ª°¾`ÒÆØM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79713189 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321387710771200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326341756941824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307570224530432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315982857297920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843720050308712e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4953941187072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142726037 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315982874337280 trainer/global_step‚8399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4953941187072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142736030 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4954029131264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63012848 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326341756941824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321387727810560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79723180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38717332 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117751586816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57771612 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302452300840960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57771382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63012850 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79713189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_b­Sh® atch_end/requested_bytes.small_pool.peak‚ 298572143 _runtime‚3946.474942336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63012848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142736030 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4954029131264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326341651958272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19054280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚320919470363596 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79723180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _step‚680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63012850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19054053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142726037 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321387727810560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79723180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38717332 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117751586816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57771612 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302452300840960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57771382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63012850 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79713189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _runtime‚3946.474942336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63012848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142736030 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4954029131264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326341651958272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19054280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚320919470363596 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79723180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _step‚680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63012850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19054053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142726037 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321387727810560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321387710771200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326341756941824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307570224530432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315982857297920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843720050308712e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4953941187072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142726037 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315982874337280 trainer/global_step‚8399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4953941187072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142736030 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4954029131264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¨ÂWi2È- _wandbruntime‚3946  _runtime‚3946´ŒmCÉ 1loss/FBt2/assignment_loss‚0.01574574833782171 1loss/SRqqt1/detection_loss‚0.8838755422838436 1loss/SRqqt2/detection_loss‚0.6169288590844313 /loss/FBt1/detection_loss‚0.5436259952738233 _step‚681 /loss/FRt1/detection_loss‚0.5730225373932766 0loss/FBt2/detection_loss‚0.29050776773633863 _runtime‚3946.476031422 0loss/FRt1/assignment_loss‚1.5783794149756432 .loss/FRt2/detection_loss‚1.437509823032544 trainer/global_step‚8399 0loss/FRt2/assignment_loss‚0.6821710664270207 1loss/SRqqt1/assignment_loss‚0.574340474286891 3loss/FBt1/assignment_loss‚0.0015708586004892125 %loss/total_loss‚7.715620116024108 epoch‚10 $ _timestamp‚1.784372005032486e+09 1loss/SRqqt2/assignment_loss‚0.517942028591986w>ű ʆ 0loss/FRt1/assignment_loss‚1.5783794149756432 .loss/FRt2/detection_loss‚1.437509823032544 trainer/global_step‚8399 0loss/FRt2/assignment_loss‚0.6821710664270207 1loss/SRqqt1/assignment_loss‚0.574340474286891 3loss/FBt1/assignment_loss‚0.0015708586004892125 %loss/total_loss‚7.715620116024108 epoch‚10 $ _timestamp‚1.784372005032486e+09 1loss/SRqqt2/assignment_loss‚0.517942028591986 1loss/FBt2/assignment_loss‚0.01574574833782171 1loss/SRqqt1/detection_loss‚0.8838755422838436 1loss/SRqqt2/detection_loss‚0.6169288590844313 /loss/FBt1/detection_loss‚0.5436259952738233 _step‚681 /loss/FRt1/detection_loss‚0.5730225373932766 0loss/FBt2/detection_loss‚0.29050776773633863 _runtime‚3946.476031422©ÊŽÞáœË‚(:‘ ²¶íÒÕÂêŠ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡)“âÌ‚(:× ²¶íÒ™Îö disk.dm-0.in‚ 5358.23046875$ memory_percent‚99.24471378326416 network.sent‚ 1.1364606e+07# disk.nvme0c0n1.in‚ 5358.23046875 disk.sda1.out‚0 cpu‚26.82891003131105 disk.sda.in‚0.109375 disk.drbd2165.in‚0" disk.dm-0.out‚6175.43896484375( proc.memory.availableMB‚ 123.74609375$ disk./.usageGB‚590.2638969421387! disk.nvme0n1.in‚ 5358.23046875 disk.sda2.in‚0 disk.dm-1.out‚ 90.51953125# proc.memory.rssMB‚ 2700.05078125* proc.memory.percent‚16.479802131652832 proc.cpu.threads‚17 network.recv‚ 1.265091e+06 disk.rbd1.in‚0% disk.nvme0n1.out‚6175.43896484375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0.109375 disk.sda.out‚ 90.484375 disk.rbd1.out‚ 9.51953125 disk.sda3.in‚0.109375' disk.nvme0c0n1.out‚6175.43896484375 disk.sda2.out‚0) disk./.usagePercent‚39.61310538818443 disk.sda3.out‚ 90.51953125 disk.drbd2165.out‚0 disk.rbd0.in‚0ž^Nž2Í- _wandbruntime‚3968  _runtime‚3968yneÝÇÎÁ % _timestamp‚1.7843720270526118e+09 _runtime‚3968.496487652 _step‚682 & lr-AdamW/pg1‚0.0007710590543653341 & lr-AdamW/pg2‚0.0007710590543653341 trainer/global_step‚8449J3zÌÏÆ _runtime‚3968.496487652 _step‚682 & lr-AdamW/pg1‚0.0007710590543653341 & lr-AdamW/pg2‚0.0007710590543653341 trainer/global_step‚8449 % _timestamp‚1.7843720270526118e+09ªav½x2Ð- _wandbruntime‚3968  _runtime‚3968W6JìÔ'ÑÎO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309309726648832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323206148018176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 % _timestamp‚1.7843720270537336e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63366677 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58099192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4964426684382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58098960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚323206148018176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328188020070912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143546466 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5146591578624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63366677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80179787 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63366679 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4981959989760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80179787 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19162144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 _step‚683 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143536487 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63366679 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5146761591296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322734430752734 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38937045 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322734327576284 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317769987028992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚8449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4981959989760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38937048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4981872052736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317770004068352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328188020070912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0L\#(ÒÓO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 _step‚683 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143536487 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63366679 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5146761591296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322734430752734 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38937045 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322734327576284 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317769987028992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚8449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4981959989760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38937048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4981872052736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317770004068352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328188020070912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309309726648832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323206148018176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor*|Ö±.on_train_batch_start/segment.all.allocated‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 % _timestamp‚1.7843720270537336e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63366677 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58099192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4964426684382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58098960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚323206148018176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328188020070912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143546466 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5146591578624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63366677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80179787 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63366679 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4981959989760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80179787 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19162144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450«$22Ó- _wandbruntime‚3968  _runtime‚3968³BàÝ&Ô×M EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143562982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19164065 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58105829 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328225805378048 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80189082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80179091 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143552989 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38941529 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5147356141056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63373900 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63373898 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58105594 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309345232713216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317806476103680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63373898 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323243276021760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317806459064320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8449 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 $ _timestamp‚1.784372027394282e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328225805378048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4964994006024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3968.838366224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38941532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304197851668480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322771470109704 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322771366933142 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4982529356288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _step‚684 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143562982 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4982441412096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143552989 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63373900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328225700394496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323243258982400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323243258982400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80179091 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19164297 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4982529356288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4982441412096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309345029415424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚323243276021760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5147177746944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304197876572160ÿ&â&ÕÜM BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 $ _timestamp‚1.784372027394282e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328225805378048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4964994006024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3968.838366224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38941532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304197851668480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322771470109704 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322771366933142 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4982529356288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 _step‚684 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143562982 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4982441412096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143552989 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63373900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328225700394496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323243258982400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323243258982400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80179091 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19164297 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4982529356288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4982441412096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309345029415424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚323243276021760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5147177746944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304197876572160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143562982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19164065 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58105829 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328225805378048 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80189082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80179091 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143552989 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38941529 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5147356141056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63373900 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63373898 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58105594 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309345232713216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317806476103680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63373898 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323243276021760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317806459064320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8449¬‰˜í2Ö- _wandbruntime‚3968  _runtime‚3968úØ ×ƒ _runtime‚3968.839668442 3loss/SRqqt1/assignment_loss‚0.48785584252839664 1loss/FBt1/assignment_loss‚0.39137656720459885 /loss/FBt1/detection_loss‚0.6311300681637866 /loss/FBt2/detection_loss‚0.4494449839380289 /loss/FRt2/detection_loss‚1.4514951048699838 0loss/SRqqt2/detection_loss‚0.695340918085431 $ _timestamp‚1.784372027396134e+09 0loss/FRt2/assignment_loss‚0.7279151744682696 3loss/SRqqt2/assignment_loss‚0.46102518684908067 2loss/FBt2/assignment_loss‚0.005877865635722081 1loss/SRqqt1/detection_loss‚0.7024820506163136 trainer/global_step‚8449 _step‚685 0loss/FRt1/assignment_loss‚1.7239988620174862 /loss/FRt1/detection_loss‚0.5620364917894286 %loss/total_loss‚8.289979116166528 epoch‚10Š;¼ú؈ 3loss/SRqqt1/assignment_loss‚0.48785584252839664 1loss/FBt1/assignment_loss‚0.39137656720459885 /loss/FBt1/detection_loss‚0.6311300681637866 /loss/FBt2/detection_loss‚0.4494449839380289 /loss/FRt2/detection_loss‚1.4514951048699838 0loss/SRqqt2/detection_loss‚0.695340918085431 $ _timestamp‚1.784372027396134e+09 0loss/FRt2/assignment_loss‚0.7279151744682696 3loss/SRqqt2/assignment_loss‚0.46102518684908067 2loss/FBt2/assignment_loss‚0.005877865635722081 1loss/SRqqt1/detection_loss‚0.7024820506163136 trainer/global_step‚8449 _step‚685 0loss/FRt1/assignment_loss‚1.7239988620174862 /loss/FRt1/detection_loss‚0.5620364917894286 %loss/total_loss‚8.289979116166528 epoch‚10 _runtime‚3968.839668442­ÍÿS‘Ù‚(:† Á¶íÒò»· gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚165.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬AælÝÚ‚(:Ò Á¶íÒ¶ø’ disk.sda1.out‚0 disk.dm-1.out‚ 91.0546875 disk.sda.out‚ 91.0546875 disk.drbd2165.out‚0$ memory_percent‚99.25029277801514( proc.memory.availableMB‚ 122.83203125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 5358.23046875$ disk.nvme0n1.out‚6178.2197265625 disk.dm-0.in‚ 5358.23046875 disk.sda3.out‚ 91.0546875 disk.drbd2165.in‚0) proc.memory.percent‚16.48256778717041 disk.sda.in‚0.109375 disk.rbd1.in‚0 disk.rbd1.out‚ 9.58203125 disk.sda2.in‚0 network.sent‚ 1.1414575e+07 network.recv‚ 1.269274e+06 disk.rbd0.out‚0 disk.sda1.in‚0! disk.dm-0.out‚6178.2197265625& disk.nvme0c0n1.out‚6178.2197265625 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375# proc.memory.rssMB‚ 2700.50390625# disk.nvme0c0n1.in‚ 5358.23046875 disk.sda2.out‚0) disk./.usagePercent‚39.61314046121662$ disk./.usageGB‚590.2644195556641 disk.rbd0.in‚0 cpu‚26.83089404245913¥Å’—‘Û‚(:† жíÒƒ§€‹ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06|ÓÜ‚(:È жíÒÛøòŒ disk.sda2.in‚0 disk.dm-0.in‚ 5358.76171875 disk.sda3.out‚ 93.08984375 disk.drbd2165.in‚0& proc.memory.availableMB‚ 122.390625 cpu‚26.833499683731063" disk.nvme0c0n1.out‚ 6195.796875$ memory_percent‚99.25298690795898 network.recv‚ 1.273547e+06 disk.sda.out‚ 93.08984375! disk.nvme0n1.in‚ 5358.76171875) disk./.usagePercent‚39.61764388975144 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 5358.76171875) proc.memory.percent‚16.48256778717041 disk.sda.in‚0.109375 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 93.08984375 disk.dm-0.out‚ 6195.796875 disk.drbd2165.out‚0 disk.rbd1.out‚ 9.58203125 disk.nvme0n1.out‚ 6195.796875 network.sent‚ 1.1477221e+07$ disk./.usageGB‚590.3315238952637 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0.109375# proc.memory.rssMB‚ 2700.50390625RDL2Ý- _wandbruntime‚3991  _runtime‚3991b1óòÇÞÁ % _timestamp‚1.7843720495566773e+09 _runtime‚3991.000563825 _step‚686 & lr-AdamW/pg1‚0.0007700810969527731 & lr-AdamW/pg2‚0.0007700810969527731 trainer/global_step‚8499fEs¾ÌßÆ trainer/global_step‚8499 % _timestamp‚1.7843720495566773e+09 _runtime‚3991.000563825 _step‚686 & lr-AdamW/pg1‚0.0007700810969527731 & lr-AdamW/pg2‚0.0007700810969527731®Òá@2à- _wandbruntime‚3991  _runtime‚3991¨–ØÜ áÍO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚324586430498842 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325061696229376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80635712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325061713268736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330072173483520 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19271308 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80635712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚3991.001757528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305908540788736 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 YEDeviceStatsMoniY” ÷tor.on_train_batch_start/allocated_bytes.all.allocated‚330072173483520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010372277760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚324586327322392 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330072068507136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63727727 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80645689 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80645689 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39161245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63727729 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319593588795392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19271546 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4992738527000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144363439 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5176198543360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58432553 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992824664090 trainer/global_step‚8499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39161248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5176026433536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144373418 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319593605834752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330072068507136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325061713268736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚687 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010372277760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325061696229376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1³SbØ'âÒO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚324586430498842 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325061696229376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80635712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325061713268736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330072173483520 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19271308 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80635712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚3991.001757528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305908540788736 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚330072173483520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010372277760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚324586327322392 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330072068507136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63727727 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80645689 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80645689 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39161245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63727729 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319593588795392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19271546 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4992738527000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144363439 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5176198543360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58432553 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992824664090 trainer/global_step‚8499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39161248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5176026433536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144373418 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319593605834752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330072068507136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325061713268736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚687 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010372277760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325061696229376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504¯b¥x 2ã- _wandbruntime‚3991  _runtime‚39915ÔZßÞ&äØM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144389934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305943452303360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010941637120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80654984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58439355 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚330109853814272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010941637120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39165729 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5176619862016 _step‚688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144379941 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4993305848530 _runtime‚3991.340481724 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011029581312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144389934 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305943427399680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993391985732 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80644993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19273403 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325098824232960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚8499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311120242170880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58439132 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324623469855812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319630077870080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚330109853814272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5176789867520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63734950 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63734948 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19273623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843720498963492e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144379941 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011029581312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843720498963492e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144379941 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011029581312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144389934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305943452303360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010941637120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80654984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58439355 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚330109853814272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010941637120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39165729 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5176619862016 _step‚688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144379941 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4993305848530 _runtime‚3991.340481724 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011029581312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144389934 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305943427399680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993391985732 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80644993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19273403 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325098824232960 4.DeviceStatsMonitor.on_train_batch_end/nu\¼$åm_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚8499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311120242170880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58439132 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324623469855812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319630077870080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚330109853814272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5176789867520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63734950 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63734948 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19273623 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5021222643798 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81111591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331956116943360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19380382 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307654091616256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5038960439808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5038960439808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39385445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5205486379520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321417207601152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4013.304810302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326917244440576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331956221919744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚691 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81101614 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58765599 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19380154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326438327068500 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39385448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326917244440576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58765830 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5205654295040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 % _timestamp‚1.7843720718608153e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64088777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326917261479936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307654116519936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64088779 trainer/global_step‚8549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5038872502784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64088777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 7524474882»ÁÙ'ðÓO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331956221919744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331956116943360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64088779 trainer/global_step‚8549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5038872502784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64088777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312859577995776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5021136506708 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64088779 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81111591 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145190391 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5021222643798 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81111591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331956116943360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19380382 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307654091616256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5038960439808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5038960439808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39385445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5205486379520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321417207601152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4013.304810302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326917244440576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331956221919744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚691 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81101614 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58765599 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19380154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326438327068500 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39385448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326917244440576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58765830 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5205654295040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 % _timestamp‚1.7843720718608153e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64088777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326917261479936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307654116519936 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145206893 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStat`bA,ÄsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145216886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81120886 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58772189 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326954355404800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5021789965440 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331993902250496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326475366425358 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326954355404800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5039441862144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321453679636480 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64095998 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843720722004082e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5039441862144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5206074008064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326954372444160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58772412 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312895077138944¡À;‚ã&óÝM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326954355404800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5021789965440 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331993902250496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326475366425358 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326954355404800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5039441862144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321453679636480 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64095998 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843720722004082e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5039441862144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5206074008064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326954372444160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58772412 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312895077138944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145216886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚326954372444160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19382480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331993797266944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64095998 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5039529806336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331993902250496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307689028034560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145206893 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145216886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81120886 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58772189 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936´bõ’ 2ô- _wandbruntime‚4013  _runtime‚4013–Üä¼ õ† /loss/FRt2/detection_loss‚1.4222634125278513 1loss/SRqqt1/detection_loss‚0.9127125011603688 epoch‚10 _runtime‚4013.645666607 3loss/FBt2/assignment_loss‚0.0010909459601116112 1loss/SRqqt2/detection_loss‚0.6513311410396699 /loss/FBt2/detection_loss‚0.5166663485900338 2loss/SRqqt1/assignment_loss‚0.5280278186060392 3loss/SRqqt2/assignment_loss‚0.44687627213905445 % _timestamp‚1.7843720722021277e+09 _step‚693 4loss/FBt1/assignment_loss‚0.00022767782902544762 .loss/FBt1/detection_loss‚0.565747880102957 %loss/total_loss‚7.821905992130615 trainer/global_step‚8549 /loss/FRt1/assignment_loss‚1.655367234858071 0loss/FRt2/assignment_loss‚0.5838574568430582 /loss/FRt1/detection_loss‚0.5377373024743745ܦš<ö‹ 4loss/FBt1/assignment_loss‚0.00022767782902544762 .loss/FBt1/detection_loss‚0.565747880102957 %loss/total_loss‚7.821905992130615 trainer/global_step‚8549 /loss/FRt1/assignment_loss‚1.655367234858071 0loss/FRt2/assignment_loss‚0.5838574568430582 /loss/FRt1/detection_loss‚0.5377373024743745 /loss/FRt2/detection_loss‚1.4222634125278513 1loss/SRqqt1/detection_loss‚0.9127125011603688 epoch‚10 _runtime‚4013.645666607 3loss/FBt2/assignment_loss‚0.0010909459601116112 1loss/SRqqt2/detection_loss‚0.6513311410396699 /loss/FBt2/detection_loss‚0.5166663485900338 2loss/SRqqt1/assignment_loss‚0.5280278186060392 3loss/SRqqt2/assignment_loss‚0.44687627213905445 % _timestamp‚1.7843720722021277e+09 _step‚693µŽÂ1‘÷‚(:† î¶íÒÌÌŸ gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ$¸Úø‚(:Ï î¶íÒ³õ© disk.sda1.in‚0 disk.dm-1.out‚ 94.015625" disk.dm-0.out‚6208.40869140625# memory_percent‚99.2513656616211 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 5428.18359375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6208.40869140625 disk.sda2.in‚0# proc.memory.rssMB‚ 2700.50390625 disk.sda.in‚0.109375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5428.18359375% disk.nvme0n1.out‚6208.40869140625 cpu‚26.837294312712135 network.sent‚ 1.1557285e+07) disk./.usagePercent‚39.61766104225624 disk.sda3.out‚ 94.015625 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.sda.out‚ 94.015625 disk.rbd1.out‚ 9.64453125 disk.rbd0.in‚0) proc.memory.percent‚16.48256778717041 network.recv‚ 1.280423e+06 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 5428.18359375$ disk./.usageGB‚590.3317794799805 disk.rbd1.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 122.65625(*]œù‚(:‘ ý¶íÒ‘žÔ‹ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öýýVàú‚(:Õ ý¶íÒž‡ËŒ' disk.nvme0c0n1.out‚6220.48486328125 disk.sda.out‚ 96.09765625% disk.nvme0n1.out‚6220.48486328125 disk.sda2.in‚0$ memory_percent‚99.25098419189453# proc.memory.rssMB‚ 2700.50390625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.1607245e+07 network.recv‚ 1.28471e+06 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.in‚0$ disk./.usageGB‚590.3981475830078 disk.sda3.out‚ 96.09765625 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0.109375) proc.memory.percent‚16.48256778717041) disk./.usagePercent‚39.62211506133695 disk.dm-0.in‚ 5428.18359375" disk.dm-0.out‚6220.48486328125 cpu‚26.839532737497702 disk.dm-1.out‚ 96.09765625# disk.nvme0c0n1.in‚ 5428.18359375 disk.rbd1.out‚ 9.70703125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5428.18359375 disk.sda2.out‚0% proc.memory.availableMB‚ 122.71875˜­ 2û- _wandbruntime‚4035  _runtime‚4035u6ÇüÁ & lr-AdamW/pg2‚0.0007681085697655128 trainer/global_step‚8599 % _timestamp‚1.7843720943076797e+09 _runtime‚4035.751606998 _step‚694 & lr-AdamW/pg1‚0.0007681085697655128[<ÂÌýÆ trainer/global_step‚8599 % _timestamp‚1.7843720943076797e+09 _runtime‚4035.751606998 _step‚694 & lr-AdamW/pg1‚0.0007681085697655128 & lr-AdamW/pg2‚0.0007681085697655128¶›Ô2þ- _wandbruntime‚4035  _runtime‚4035ÈÚœ Ó'ÿÍO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 trainer/global_step‚8599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚4035.752878689 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81567516 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333840165379584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146027322 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333840165379584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39609645 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64449829 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309399667347456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59099013 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 $ _timestamp‚1.784372094308866e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81577493 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328772809691136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚695 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333840270355968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314634866121728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328290429991058 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19489365 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328290326814608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323240809367552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333840270355968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39609648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067372727808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64449829 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81577493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5049534486416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333840165379584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146027322 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333840165379584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39609645 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64449829 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309399667347456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59099013 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 $ _timestamp‚1.784372094308866e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81577493 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328772809691136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚695 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333840270355968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314634866121728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328290429991058 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19489365 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328290326814608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323240809367552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333840270355968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39609648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067372727808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146·¼ÞG2- _wandbruntime‚4036  _runtime‚4036úEÞ&‚ØM _runtime‚4036.094974475 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81586788 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328809920655360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59105352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19491453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309434578862080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309434603765760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64457048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64457050 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328809903616000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81576797 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328327469348028 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5050187945148 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146043838 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333877950686720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328327366171466 trainer/global_step‚8599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333877845703168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚323277281402880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146033845 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5067942087168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39614132 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5235592733696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5235762739200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81576797 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333877950686720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323277264363520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328809920655360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81586788 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314670366504960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328809903616000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59105585 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314670171595776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843720946508558e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64457048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5050101807946 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64457050 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146043838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ôBã&ƒÝM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39614129 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59105352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19491453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309434578862080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309434603765760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64457048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64457050 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328809903616000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81576797 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328327469348028 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5050187945148 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146043838 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333877950686720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328327366171466 trainer/global_step‚8599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333877845703168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚323277281402880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146033845 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5067942087168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39614132 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5235592733696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5235762739200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81576797 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333877950686720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323277264363520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328809920655360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81586788 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314670366504960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328809903616000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59105585 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314670171595776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843720946508558e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64457048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5050101807946 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64457050 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146043838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4036.094974475 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81586788 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328809920655360 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146844295 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64810879 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5095960889856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64810877 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59432594 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311145243078656 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚8649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39833845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325064394094592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335724213815808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5095872952832 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64810877 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335724318792192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39833848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19598746 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330628357902336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5095960889856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5078018603214 _step‚699 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146854274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 _runtime‚4057.993788805 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784372116549757e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316409919553024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316409724636672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330628340862976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59432365 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5264651570688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19598520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5264481558016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5095872952832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452Ô¬œuØ'ŽÒO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330628340862976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82043395 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330142429737166 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64810879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330142326560716 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64810877 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335724318792192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39833848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19598746 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330628357902336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5095960889856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5078018603214 _step‚699 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146854274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 _runtime‚4057.993788805 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784372116549757e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316409919553024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316409724636672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330628340862976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59432365 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5264651570688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19598520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5264481558016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5095872952832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82043395 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335724318792192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82033418 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330628357902336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82033418 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5077932466124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146844295 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64810879 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5095960889856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64810877 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59432594 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311145243078656 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚8649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39833845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325064394094592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335724213815808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5095872952832 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64818100 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316445419588096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316445226776064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82052690 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19600878 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5265240091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5265072182784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5078585924856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335761999122944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59439210 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82042699 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330665451827200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146870790 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146870790 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚4058.332798039 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64818098 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82042699 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330179365917574 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59438993 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146860797 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311180179496960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚8649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330665468866560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3FÞQÿ> ‘ÝM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64818100 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316445419588096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316445226776064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82052690 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19600878 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5265240091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5265072182784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838329 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5078585924856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 P8DeviceStatsMonitor.on_train_batch_end/Ò‚¯Ø¥num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335761999122944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59439210 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82042699 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330665451827200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146870790 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146870790 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚4058.332798039 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64818098 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82042699 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330179365917574 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59438993 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146860797 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311180179496960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚8649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330665468866560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5096442312192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64818098 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚214 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335761894139392 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146860797 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64818100 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312890843713536 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147671247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331994429483274 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 _runtime‚4080.401919986 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65171927 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82509297 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326888012900352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19707724 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40058045 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337608367228416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19707941 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65171929 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82499320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59765769 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337608262252032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65171927 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337608262252032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5124373177856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5124461114880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82509297 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326887995860992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚8699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332483889074176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332483906113536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚217 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5106416582922 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0NºØ'žÒO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337608367228416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19707941 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65171929 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82499320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59765769 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337608262252032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65171927 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337608262252032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5124373177856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5124461114880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82509297 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326887995860992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚8699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332483889074176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332483906113536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚217 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5106416582922 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312890843713536 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147671247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331994429483274 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 _runtime‚4080.401919986 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65171927 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82509297 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326888012900352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19707724 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40058045 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143¿‰ëue2Ÿ- _wandbruntime‚4080  _runtime‚4080Üø Éõ ØM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312925755228160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147687749 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19710057 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337646047559168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82508601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332521017077760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318220505242112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19709844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59772589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147697742 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147687749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326924467896320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337646047559168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65179150 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332521017077760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312925730324480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5125030481408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65179150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59772373 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473(L®iã&¡ÝM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65179148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40062529 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 % _timestamp‚1.7843721392986672e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65179148 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326924484935680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82518592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40062532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332031365663682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312925755228160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147687749 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19710057 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337646047559168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82508601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332521017077760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318220505242112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19709844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59772589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147697742 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147687749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326924467896320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337646047559168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65179150 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332521017077760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312925730324480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5125030481408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65179150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59772373 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339492310688256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314636419444736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40282248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314636394541056 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚334339437285376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5323499538944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333846429229382 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148508178 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82965222 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5152873402880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5152873402880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4102.655868853 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82965222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5152961339904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19817132 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5134814562630 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339492415664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65532977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82975199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319960091093504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334339454324736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚707 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40282245 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148498199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19816912 $ _timestamp‚1.784372161211861e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚334339437285376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328711597627392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚334339454324736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5152961339904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60099157 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚8749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333846326052932 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339492415664640 ]ODeviceStatsMonitor.on_train_batch_start/inalÿn9”ctive_split_bytes.small_pool.current‚ 172109824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328711614666752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60099380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Œ}>Ø'¬ÒO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65532979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65532979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319959894080000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339492310688256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314636419444736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40282248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314636394541056 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚334339437285376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5323499538944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333846429229382 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148508178 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82965222 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5152873402880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5152873402880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4102.655868853 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82965222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5152961339904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19817132 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5134814562630 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339492415664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65532977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82975199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319960091093504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334339454324736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚707 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40282245 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148498199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19816912 $ _timestamp‚1.784372161211861e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚334339437285376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328711597627392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚334339454324736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5152961339904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60099157 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚8749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333846326052932 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339492415664640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328711614666752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60099380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5134728425540 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339492310688256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82975199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5323671648768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65532977ÃK 2­- _wandbruntime‚4102  _runtime‚4102I…þ:Ý&®×M cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314671330959360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148524694 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65540200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5153530706432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334376565288960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5153442762240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60106000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19819268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314671306055680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65540200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65540198 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82974503 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334376548249600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339530095995392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60105779 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82984494 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚328748086702080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333883365409790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40286732 h½â&¯ÜM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65540198 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5324264068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148514701 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148514701 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5135295747070 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319995595027968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚334376548249600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5135381884272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚708 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339530095995392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚334376565288960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82984494 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19819050 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 % _timestamp‚1.7843721615498576e+09 _runtime‚ 4102.99392886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚328748086702080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333883365409790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40286732 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314671330959360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148524694 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65540200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5153530706432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334376565288960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5153442762240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60106000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19819268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314671306055680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65540200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65540198 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82974503 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334376548249600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339530095995392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60105779 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82984494 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341376464100864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330535216433152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181373627904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149335130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65894027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321735002057728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65894027 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83431124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40506448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330535199393792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336195002535936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65894029 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83431124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5181461564928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60433961 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149325151 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335698428975490 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181461564928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83441101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336194985496576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336195002535936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316381970272256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316381995175936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5163212542338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5353031785472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341376359124480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341376464100864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181373627904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19927513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19927291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5353205992448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5163126405248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83441101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚8799 _runtime‚4125.103242804 _step‚711 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341376359124480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60433736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚335698325799040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024RÛcOÙ'¼ÓO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341376359124480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60433736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚335698325799040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40506445 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341376464100864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330535216433152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181373627904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149335130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65894027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321735002057728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65894027 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83431124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40506448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330535199393792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336195002535936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65894029 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83431124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5181461564928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60433961 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149325151 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335698428975490 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181461564928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83441101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336194985496576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336195002535936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316381970272256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316381995175936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5163212542338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5353031785472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341376359124480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341376464100864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181373627904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19927513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19927291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5353205992448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5163126405248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83441101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚8799 _runtime‚4125.103242804 _step‚711 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0dzÇi<2½- _wandbruntime‚4125  _runtime‚4125›I߀"$¾ØM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83440405 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341414144431616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336232096460800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341414144431616 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60440270 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149351646 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65901248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336232113500160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 % _timestamp‚1.7843721840009305e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149351646 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321770494661632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83440405 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182030931456 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336232096460800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316416881786880 _step‚712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19929574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335735468332460 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149341653 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83450396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40510932 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330571671429120 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316416906690560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335735365155898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336232113500160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321770704250880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5163693726778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40510929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19929341 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚Ü€[1¼ 5181942987264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149341653 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83450396 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚8799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5353612874752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330571688468480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚08e6^ã&¿ÝM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40510929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19929341 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5181942987264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149341653 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83450396 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 trainer/global_step‚8799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5353612874752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330571688468480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83440405 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341414144431616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336232096460800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341414144431616 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60440270 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149351646 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65901248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193719296 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336232113500160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 % _timestamp‚1.7843721840009305e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149351646 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321770494661632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83440405 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182030931456 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336232096460800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316416881786880 _step‚712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19929574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335735468332460 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149341653 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83450396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40510932 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330571671429120 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316416906690560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335735365155898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336232113500160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321770704250880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5163693726778ÈÀ*ø?2À- _wandbruntime‚4125  _runtime‚4125^™€€ Áƒ /loss/FRt2/assignment_loss‚0.763246120490464 1loss/SRqqt2/assignment_loss‚0.500163361217876 2loss/FBt2/assignment_loss‚0.012112092444847159 /loss/FRt2/detection_loss‚1.5834896707714632 /loss/FRt1/detection_loss‚0.5893374049333985 %loss/total_loss‚8.273335097086877 epoch‚10 _runtime‚4125.446064612 0loss/FRt1/assignment_loss‚1.6026393322436983 3loss/FBt1/assignment_loss‚0.0005435712626681916 1loss/SRqqt1/detection_loss‚0.7585936542699471 /loss/FBt2/detection_loss‚0.6354721217377421 2loss/SRqqt1/assignment_loss‚0.6468245572190078 1loss/SRqqt2/detection_loss‚0.5801699456045569 /loss/FBt1/detection_loss‚0.6007432648912072 trainer/global_step‚8799 % _timestamp‚1.7843721840025258e+09 _step‚713ObˆAˆ 2loss/SRqqt1/assignment_loss‚0.6468245572190078 1loss/SRqqt2/detection_loss‚0.5801699456045569 /loss/FBt1/detection_loss‚0.6007432648912072 trainer/global_step‚8799 % _timestamp‚1.7843721840025258e+09 _step‚713 /loss/FRt2/assignment_loss‚0.763246120490464 1loss/SRqqt2/assignment_loss‚0.500163361217876 2loss/FBt2/assignment_loss‚0.012112092444847159 /loss/FRt2/detection_loss‚1.5834896707714632 /loss/FRt1/detection_loss‚0.5893374049333985 %loss/total_loss‚8.273335097086877 epoch‚10 _runtime‚4125.446064612 0loss/FRt1/assignment_loss‚1.6026393322436983 3loss/FBt1/assignment_loss‚0.0005435712626681916 1loss/SRqqt1/detection_loss‚0.7585936542699471 /loss/FBt2/detection_loss‚0.6354721217377421É/5íΑÂ(:† æ·íÒŒÁ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚197.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.5456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GbßÄ‚(:Ô æ·íÒ‰ÖŽ disk.dm-1.in‚0.109375 disk.sda.in‚0.109375 disk.sda.out‚ 100.3046875 disk.nvme0n1.in‚ 5569.640625! disk.nvme0c0n1.in‚ 5569.640625 disk.rbd0.in‚0 network.sent‚ 1.1886254e+07 disk.sda2.in‚0 disk.sda3.out‚ 100.3046875 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 9.95703125) disk./.usagePercent‚39.63115289531823 disk.dm-1.out‚ 100.3046875 network.recv‚ 1.311825e+06 disk.sda2.out‚0 disk.dm-0.in‚ 5569.640625 disk.sda3.in‚0.109375' disk.nvme0c0n1.out‚6522.89892578125% disk.nvme0n1.out‚6522.89892578125( proc.memory.availableMB‚ 121.48828125 cpu‚26.864536416383938$ disk./.usageGB‚590.5328178405762 disk.sda1.out‚0$ memory_percent‚99.25849437713623# proc.memory.rssMB‚ 2701.32421875" disk.dm-0.out‚6522.89892578125 disk.drbd2165.in‚0* proc.memory.percent‚16.487574577331543 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0ƒŒu2Å- _wandbruntime‚4151  _runtime‚4151`í°µ=Ư{ 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6573859453201294 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5548439025878906 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6648545265197754 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960920214653015 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18899103999137878 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.422839492559433 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982757031917572 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47483426332473755 +jet/accuracy_4_of_4‚0.01071428507566452 7validation_average_jet_accuracy‚0.34956833720207214 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895633459091187 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19995421171188354 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1652442067861557 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_5‚1 _step‚714 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4932958483695984 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1786791980266571 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7353746891021729 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14145322144031525 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.048356108367443085 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7410603165626526 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7527540922164917 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6599258184432983 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913784861564636 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36583641171455383 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5252604484558105 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13675634562969208 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15927572548389435 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1963217854499817 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7080194354057312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16858024895191193 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37098973989486694 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6295478940010071 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8495727181434631 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8434210419654846 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6295144557952881 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23487965762615204 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.661145031452179 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8918343782424927 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9236235618591309 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13906249403953552 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6552455425262451 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9312834143638611 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6326810717582703 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.730213463306427 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6295144557952881 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03351619839668274 *jet/accuracy_2_of_3‚0.5303664207458496 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43573901057243347 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6109222769737244 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8774617314338684 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09388667345046997 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 )loss/val_total_loss‚8.019761277884843 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6117267608642578 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5920607447624207 +particle/sensitivity‚0.6198879480361938 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7407521605491638 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04555976763367653 particle/accuracy_3_of_3‚1 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.769658088684082 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 % _timestamp‚1.7843722096844933e+09 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7028676271438599 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.707316517829895 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13292589783668518 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14966687560081482 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16579477488994598 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46425819396972656 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6357945203781128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6284722685813904 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12407734245061874 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.049429696053266525 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.845086932182312 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7073999643325806 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18780839443206787 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6696589589118958 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12476383149623871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14970891177654266 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3051277995109558 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957407712936401 +jet/accuracy_3_of_3‚0.07595473527908325 epoch‚10 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9895644187927246 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014239595271646976 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24779653549194336 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7612468004226685 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968363642692566 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9965643286705017 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.765157163143158 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8434210419654846 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16603398323059082 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0477287694811821 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9971349835395813 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17011211812496185 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7848411798477173 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1496589183807373 )jet/accuracy_2_of_4‚0.734869122505188 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.670526921749115 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01728079468011856 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7848411798477173 +particle/specificity‚0.9066770672798157 'particle/f_score‚0.6321280598640442 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04555976763367653 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957658648490906 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.020949779078364372 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8802787065505981 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4827621579170227 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 _runtime‚4151.128978954 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1632591038942337 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17907841503620148 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16579477488994598 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.697812557220459 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7353090643882751 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2878853678703308 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5532244443893433 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6115504503250122 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11633406579494476 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03142008185386658 *jet/accuracy_2_of_2‚0.1464499533176422 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44819754362106323 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7532158493995667 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8468672037124634 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_1_of_1‚0.30352988839149475 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9506312608718872 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6629883050918579 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.754391610622406 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00819205492734909 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6878361701965332 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948475956916809 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8539988398551941 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18012592196464539 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6346198916435242 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.049738917499780655 trainer/global_step‚8810 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6605654954910278 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6272498965263367 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6141871809959412 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6993475556373596 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9262763261795044 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.163884699344635 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9438742399215698×äúz{ Ç´{ =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14966687560081482 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16579477488994598 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46425819396972656 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6357945203781128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6284722685813904 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12407734245061874 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.049429696053266525 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.845086932182312 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7073999643325806 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18780839443206787 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6696589589118958 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12476383149623871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14970891177654266 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3051277995109558 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957407712936401 +jet/accuracy_3_of_3‚0.07595473527908325 epoch‚10 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9895644187927246 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014239595271646976 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24779653549194336 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7612468004226685 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968363642692566 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9965643286705017 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_ɬƒ?11_of_2‚0.765157163143158 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8434210419654846 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16603398323059082 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0477287694811821 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9971349835395813 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17011211812496185 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7848411798477173 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1496589183807373 )jet/accuracy_2_of_4‚0.734869122505188 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.670526921749115 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01728079468011856 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7848411798477173 +particle/specificity‚0.9066770672798157 'particle/f_score‚0.6321280598640442 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04555976763367653 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957658648490906 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.020949779078364372 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8802787065505981 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4827621579170227 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 _runtime‚4151.128978954 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1632591038942337 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17907841503620148 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16579477488994598 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.697812557220459 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7353090643882751 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2878853678703308 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5532244443893433 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6115504503250122 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11633406579494476 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03142008185386658 *jet/accuracy_2_of_2‚0.1464499533176422 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44819754362106323 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7532158493995667 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8468672037124634 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_1_of_1‚0.30352988839149475 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9506312608718872 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6629883050918579 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.754391610622406 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00819205492734909 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6878361701965332 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948475956916809 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8539988398551941 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18012592196464539 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6346198916435242 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.049738917499780655 trainer/global_step‚8810 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6605654954910278 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6272498965263367 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6141871809959412 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6993475556373596 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9262763261795044 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.163884699344635 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9438742399215698 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6573859453201294 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5548439025878906 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6648545265197754 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960920214653015 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18899103999137878 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.422839492559433 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982757031917572 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47483426332473755 +jet/accuracy_4_of_4‚0.01071428507566452 7validation_average_jet_accuracy‚0.34956833720207214 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895633459091187 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19995421171188354 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1652442067861557 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_5‚1 _step‚714 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4932958483695984 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1786791980266571 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7353746891021729 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14145322144031525 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.048356108367443085 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7410603165626526 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7527540922164917 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6599258184432983 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9913784861564636 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36583641171455383 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5252604484558105 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918777942657471 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13675634562969208 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15927572548389435 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1963217854499817 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7080194354057312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16858024895191193 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37098973989486694 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6295478940010071 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8495727181434631 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8434210419654846 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6295144557952881 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23487965762615204 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.661145031452179 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8918343782424927 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9236235618591309 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13906249403953552 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6552455425262451 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9312834143638611 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6326810717582703 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.730213463306427 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6295144557952881 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03351619839668274 *jet/accuracy_2_of_3‚0.5303664207458496 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43573901057243347 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6109222769737244 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8774617314338684 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09388667345046997 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 )loss/val_total_loss‚8.019761277884843 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6117267608642578 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5920607447624207 +particle/sensitivity‚0.6198879480361938 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7407521605491638 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04555976763367653 particle/accuracy_3_of_3‚1 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.769658088684082 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 % _timestamp‚1.7843722096844933e+09 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7028676271438599 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.707316517829895 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13292589783668518 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875ÊÂÐz‘È‚(:† õ·íÒ×΋ gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚173.108' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6216 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷5 ÜàÉ‚(:Õ õ·íÒ룚Œ disk.rbd1.in‚0# proc.memory.rssMB‚ 2710.08203125* disk./.usagePercent‚39.631141630986725! disk.nvme0c0n1.in‚ 5569.640625% disk.nvme0n1.out‚6652.98876953125 disk.sda3.out‚ 100.7265625 disk.sda.in‚0.109375( proc.memory.availableMB‚ 196.83984375 cpu‚26.876238280874627 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 100.7265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 5569.640625" disk.dm-0.out‚6652.98876953125 disk.sda3.in‚0.109375$ disk./.usageGB‚590.5326499938965' disk.nvme0c0n1.out‚6652.98876953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 100.7265625 disk.rbd1.out‚ 10.01953125 disk.sda1.in‚0$ memory_percent‚98.79858493804932* proc.memory.percent‚16.541028022766113 network.recv‚ 1.31633e+06 disk.nvme0n1.in‚ 5569.640625 disk.rbd0.out‚0 network.sent‚ 1.1936203e+07ƯË÷2Ê- _wandbruntime‚4169  _runtime‚4169àþ<ÇËÁ & lr-AdamW/pg1‚0.0007630810876624426 & lr-AdamW/pg2‚0.0007630810876624426 trainer/global_step‚8849 % _timestamp‚1.7843722280557957e+09 _runtime‚4169.499610136 _step‚715ó*œãÌÌÆ & lr-AdamW/pg2‚0.0007630810876624426 trainer/global_step‚8849 % _timestamp‚1.7843722280557957e+09 _runtime‚4169.499610136 _step‚715 & lr-AdamW/pg1‚0.0007630810876624426Ë=n¡í2Í- _wandbruntime‚4169  _runtime‚4169PQcÓ'ÎÍO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66489607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150530209 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84040604 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344238005748224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344237900771840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338524032020514 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _runtime‚ 4169.50087302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚8849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66489607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20106008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338524135196964 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344237900771840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚339011530100736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319042084331520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333316019034112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66489605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84050581 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84050581 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324441052296704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5398992868864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339011530100736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5226387710464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150540188 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333316036073472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5208099123492 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339011513061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40835228 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84050581 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84050581 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324441052296704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5398992868864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339011530100736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5226387710464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150540188 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193719296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333316036073472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5208099123492 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339011513061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40835228 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 trainer/global_step‚8849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66489607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20106008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338524135196964 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344237900771840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚339011530100736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319042084331520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333316019034112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66489605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ÌÂFR 2Ð- _wandbruntime‚4169  _runtime‚4169@ÄÃÞ&ÑØM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339048641064960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339048624025600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150556704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339048624025600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5226957069824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84059876 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338561071377372 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚333352508108800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338561174553934 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40839712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319076995846144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60947612 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84049885 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84049885 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5227045014016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339048641064960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66496826 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324476753754624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333352491069440 trainer/global_step‚8849 % _timestamp‚1.7843722283965716e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344275686078976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344275581095424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _runtime‚4169.840703365 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84059876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2+=×¾ã&ÒÝM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150546711 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66496828 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150546711 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60947612 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84049885 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84049885 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5227045014016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339048641064960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66496826 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324476753754624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333352491069440 trainer/global_step‚8849 % _timestamp‚1.7843722283965716e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344275686078976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344275581095424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _runtime‚4169.840703365 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84059876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339048641064960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339048624025600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150556704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339048624025600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5226957069824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84059876 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338561071377372 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚333352508108800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338561174553934 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40839712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319076995846144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335139637839872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335139620800512 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5254975872512 ^LDeviceStatsMonitor.on_train_batch_start/alloc¼Ëåàæated_bytes.small_pool.allocated‚ 5254975872512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20214212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66850657 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346122054184448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340376134943072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84506506 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61273640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346121949208064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340867078311936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326216107051008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5428644001792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151357161 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61273419 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346121949208064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5254887935488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326216304064512 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326216107051008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5428644001792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151357161 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61273419 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346121949208064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5254887935488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚218 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326216304064512 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335139637839872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335139620800512 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5254975872512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5254975872512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20214212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66850657 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346122054184448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340376134943072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84506506 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61273640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154ÐõÎ2à- _wandbruntime‚4192  _runtime‚4192ìg©1Þ&áØM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41063912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340413071123480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5429231977472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340904172236800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61280048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41063909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84515787 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20216139 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚8899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5255457294848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66857878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5237064424842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66857876 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5255457294848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151383656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5255545239040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84525778 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4192.393633617 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5429064069120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340413174300042 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335176109875200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346159734515200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151373663 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843722509495206e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335176092835840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61280259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20216347 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326251610742784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5429231977472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340904172236800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61280048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41063909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84515787 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20216139 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 trainer/global_step‚8899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5255457294848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66857878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5237064424842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66857876 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5255457294848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151383656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5255545239040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84525778 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4192.393633617 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5429064069120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340413174300042 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335176109875200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346159734515200 H:DeviceStatsMonitor.on_train_batch_end/alŠ~Õlocation.all.freed‚ 151373663 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843722509495206e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335176092835840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61280259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20216347 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326251610742784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 P.¿Ùç‚(:Î ¢¸íÒ§µ†Ž disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.dm-0.out‚ 7056.27734375 disk.sda3.out‚ 104.51953125 disk.rbd1.out‚ 10.08203125 network.sent‚ 1.2074089e+07 disk.sda.in‚0.109375 disk.drbd2165.in‚0 cpu‚26.88196655948182) disk./.usagePercent‚39.61362303541866 disk.dm-1.out‚ 104.51953125 disk.dm-0.in‚ 5641.109375" disk.nvme0n1.out‚ 7056.27734375( proc.memory.availableMB‚ 197.18359375 proc.cpu.threads‚17 network.recv‚ 1.327783e+06 disk.sda1.out‚0 disk.sda3.in‚0.109375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 5641.109375$ disk.nvme0c0n1.out‚ 7056.27734375$ memory_percent‚98.79648685455322# proc.memory.rssMB‚ 2710.08203125* proc.memory.percent‚16.541028022766113 disk.sda.out‚ 104.51953125 disk.rbd0.out‚0$ disk./.usageGB‚590.2716102600098 disk.sda2.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 5641.109375Œ0| 2è- _wandbruntime‚4214  _runtime‚4214n¦žPÅé¿ _runtime‚4214.460552183 _step‚723 % lr-AdamW/pg1‚0.000761031967372551 % lr-AdamW/pg2‚0.000761031967372551 trainer/global_step‚8949 % _timestamp‚1.7843722730166967e+09T›dÐÊêÄ trainer/global_step‚8949 % _timestamp‚1.7843722730166967e+09 _runtime‚4214.460552183 _step‚723 % lr-AdamW/pg1‚0.000761031967372551 % lr-AdamW/pg2‚0.000761031967372551ÓøGåG2ë- _wandbruntime‚4214  _runtime‚4214-+[_Ò'ìÌO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5283476097536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5264808945818 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚342722626523136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚336963222566912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322533235793920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342228134689180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84972408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5457989771776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚348005997644288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67211707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322533210890240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5283388160512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67211705 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84982385 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61607107 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41283625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327991391384064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚348006102620672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5458155590144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67211707 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 _step‚724 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚348006102620672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚342722609483776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61606879 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84972408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚342722609483776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20323254 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342228031512730 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5283388160512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152184113 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67211705 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚342722626523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚348005997644288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20323479 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 7524474882?„ê×'íÑO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚348006102620672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚342722609483776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61606879 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84972408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚342722609483776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20323254 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342228031512730 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5283388160512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152184113 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67211705 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚342722626523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚348005997644288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20323479 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5283476097536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5264808945818 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚342722626523136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚336963222566912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322533235793920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342228134689180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84972408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5457989771776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚348005997644288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67211707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322533210890240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5283388160512 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67211705 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84982385 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61607107 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41283625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327991391384064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚348006102620672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5458155590144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67211707 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 _step‚724ÔdÞN˜2î- _wandbruntime‚4214  _runtime‚4214!~!=ïØM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚336999694602240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61613596 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67218926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚348043782951424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152210608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328026891353600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348043782951424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284045464064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61613826 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5458580331008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84981689 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284045464064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚336999711641600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5265462404550 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41288109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20325487 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67218928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283957519872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67218928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322568122404864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚4214.806859477 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚725 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67218926 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327·#»ý¡  JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20325714 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322568147308544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41288112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5265376267348 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342265070869588 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚342759737487360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚342759720448000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚348043677967872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163714048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342265174046150 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚342759720448000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 trainer/global_step‚8949 % _timestamp‚1.7843722733627791e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84981689 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0C Šã&ðÝM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348043782951424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284045464064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61613826 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5458580331008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84981689 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284045464064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚336999711641600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5265462404550 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41288109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20325487 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67218928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283957519872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67218928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322568122404864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚4214.806859477 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚725 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67218926 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20325714 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322568147308544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41288112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5265376267348 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342265070869588 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚342759737487360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚342759720448000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚348043677967872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163714048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342265174046150 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚342759720448000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 trainer/global_step‚8949 % _timestamp‚1.7843722733627791e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84981689 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚336999694602240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61613596 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67218926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚348043782951424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152210608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328026891353600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473Õü®|»2ñ- _wandbruntime‚4214  _runtime‚4214tT$üò‚ $loss/total_loss‚8.32371209212559 epoch‚11 _runtime‚4214.808344851 0loss/FRt1/assignment_loss‚1.7347404956817627 2loss/SRqqt1/assignment_loss‚0.6504689846901449 /loss/FRt1/detection_loss‚0.5500385420339748 /loss/FRt2/detection_loss‚1.5251237211311806 /loss/FBt1/detection_loss‚0.4262550354003906 /loss/FBt2/detection_loss‚0.6144900018237267 $ _timestamp‚1.784372273364802e+09 2loss/FBt2/assignment_loss‚0.026541970465340382 1loss/SRqqt1/detection_loss‚0.8422823145480479 1loss/SRqqt2/detection_loss‚0.6722048297397454 trainer/global_step‚8949 2loss/SRqqt2/assignment_loss‚0.5329047992417958 2loss/FBt1/assignment_loss‚0.006784984958352899 _step‚726 0loss/FRt2/assignment_loss‚0.7418764124111253ëgYF ó‡ 2loss/SRqqt2/assignment_loss‚0.5329047992417958 2loss/FBt1/assignment_loss‚0.006784984958352899 _step‚726 0loss/FRt2/assignment_loss‚0.7418764124111253 $loss/total_loss‚8.32371209212559 epoch‚11 _runtime‚4214.808344851 0loss/FRt1/assignment_loss‚1.7347404956817627 2loss/SRqqt1/assignment_loss‚0.6504689846901449 /loss/FRt1/detection_loss‚0.5500385420339748 /loss/FRt2/detection_loss‚1.5251237211311806 /loss/FBt1/detection_loss‚0.4262550354003906 /loss/FBt2/detection_loss‚0.6144900018237267 $ _timestamp‚1.784372273364802e+09 2loss/FBt2/assignment_loss‚0.026541970465340382 1loss/SRqqt1/detection_loss‚0.8422823145480479 1loss/SRqqt2/detection_loss‚0.6722048297397454 trainer/global_step‚8949ÖWš[ª‘ô‚(:† ±¸íÒùÄÁŠ gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚170.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t¸„0Öõ‚(:Ë ±¸íÒË÷«Œ network.recv‚ 1.33235e+06* disk./.usagePercent‚39.621396960203406$ disk./.usageGB‚590.3874473571777 disk.rbd0.out‚0' proc.memory.availableMB‚ 196.6328125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5641.109375$ disk.nvme0c0n1.out‚ 7059.69921875 disk.drbd2165.in‚0 disk.rbd1.out‚ 10.14453125 disk.dm-1.out‚ 106.0078125 disk.dm-0.out‚ 7059.69921875 disk.sda.out‚ 106.0078125 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 5641.109375$ memory_percent‚98.79984855651855* proc.memory.percent‚16.541028022766113 disk.sda3.out‚ 106.0078125 disk.sda1.out‚0 disk.dm-1.in‚0.109375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 5641.109375 disk.sda1.in‚0 cpu‚26.884173808620954 network.sent‚ 1.2123883e+07 disk.sda3.in‚0.109375" disk.nvme0n1.out‚ 7059.69921875 disk.sda.in‚0.109375 disk.rbd1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2710.08203125…fŽ‘ö‚(:† À¸íÒà•ÆŒ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚177.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øõ!Éß÷‚(:Ô À¸íÒ«‘ìŽ disk.sda3.in‚0.109375 disk.dm-1.in‚0.109375# proc.memory.rssMB‚ 2710.08203125$ disk./.usageGB‚590.3845138549805& disk.nvme0c0n1.out‚7067.8681640625 disk.sda2.out‚0 disk.dm-1.out‚ 107.171875' proc.memory.availableMB‚ 196.7109375 network.sent‚ 1.2186595e+07 disk.sda3.out‚ 107.171875 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 10.14453125 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0n1.out‚7067.8681640625 disk.rbd0.out‚0 disk.dm-0.in‚ 5641.11328125! disk.dm-0.out‚7067.8681640625 proc.cpu.threads‚17 network.recv‚ 1.336303e+06) disk./.usagePercent‚39.62120009040957 disk.sda.in‚0.109375 disk.sda.out‚ 107.171875 disk.sda2.in‚0* proc.memory.percent‚16.541028022766113 cpu‚26.886016936203664 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 5641.11328125! disk.nvme0n1.in‚ 5641.11328125$ memory_percent‚98.79937171936035®˜ñg2ø- _wandbruntime‚4236  _runtime‚4236Ž|¡;Äù¾ $ _timestamp‚1.784372295553688e+09 _runtime‚4236.997613777 _step‚727 % lr-AdamW/pg1‚0.000759999302353542 % lr-AdamW/pg2‚0.000759999302353542 trainer/global_step‚8999‰úÉúà % lr-AdamW/pg1‚0.000759999302353542 % lr-AdamW/pg2‚0.000759999302353542 trainer/global_step‚8999 $ _timestamp‚1.784372295553688e+09 _runtime‚4236.997613777 _step‚727×Úïh-2û- _wandbruntime‚4236  _runtime‚4236#ðݲÔ'üÎO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67572755 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344578174734336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5487636418560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5487476891648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20432763 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67572755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20432535 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5311888385536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85448287 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5311976322560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349890046080512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344578157694976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41507825 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324278811525120 % _timestamp‚1.7843722955548396e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85438310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344578174734336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 trainer/global_step‚8999 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85448287 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67572757 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153021044 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344080134435288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61940591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 184430592 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324278786621440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159526912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344080031258838 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338786824333312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5311976322560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344578157694976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20432763 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67572755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20432535 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5311888385536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85448287 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5311976322560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349890046080512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344578157694976 TGDeviceStatsMoŠ“^¤©nitor.on_train_batch_start/inactive_split.large_pool.freed‚41507825 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324278811525120 % _timestamp‚1.7843722955548396e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85438310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344578174734336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 trainer/global_step‚8999 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85448287 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67572757 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153021044 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344080134435288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61940591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 184430592 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324278786621440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159526912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344080031258838 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338786824333312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5311976322560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344578157694976 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153037560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153027567 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843722958941998e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344615268659200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329801958050816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚8999 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338823296368640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67579978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324313698136064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5488235011072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5312457744896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41512312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324313723039744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5293860384258 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61947262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5312457744896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153037560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61947036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338823313408000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85447591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349927831387648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344615268659200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5488056616960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223ý·{Áã&€ÝM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85447591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349927831387648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344615268659200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5488056616960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67579976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344615285698560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41512309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20434727 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5293774247056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349927726404096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5312545689088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349927831387648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67579978 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153037560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153027567 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843722958941998e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344615268659200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329801958050816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 trainer/global_step‚8999 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338823296368640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67579978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324313698136064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5488235011072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5312457744896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41512312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324313723039744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5293860384258 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61947262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5312457744896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153037560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61947036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338823313408000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202Ùæý!`2- _wandbruntime‚4237  _runtime‚4237œx R ‚„ 2loss/SRqqt1/assignment_loss‚0.6142076963780266 3loss/SRqqt2/assignment_loss‚0.48154558012059323 3loss/FBt1/assignment_loss‚0.0052189351618034275 1loss/SRqqt1/detection_loss‚0.7315146339920963 1loss/SRqqt2/detection_loss‚0.5441440315107906 epoch‚11 _runtime‚4237.339564737 _step‚730 0loss/FRt2/assignment_loss‚0.7062758705019951 3loss/FBt2/assignment_loss‚4.666049099467098e-05 .loss/FRt2/detection_loss‚1.450018987208605 .loss/FBt1/detection_loss‚0.549287159036359 $loss/total_loss‚7.77173373046935 trainer/global_step‚8999 % _timestamp‚1.7843722958960223e+09 /loss/FRt1/assignment_loss‚1.673867900683501 /loss/FRt1/detection_loss‚0.5187349888852555 0loss/FBt2/detection_loss‚0.49687128649932927ðçÂ\ƒ‰ epoch‚11 _runtime‚4237.339564737 _step‚730 0loss/FRt2/assignment_loss‚0.7062758705019951 3loss/FBt2/assignment_loss‚4.666049099467098e-05 .loss/FRt2/detection_loss‚1.450018987208605 .loss/FBt1/detection_loss‚0.549287159036359 $loss/total_loss‚7.77173373046935 trainer/global_step‚8999 % _timestamp‚1.7843722958960223e+09 /loss/FRt1/assignment_loss‚1.673867900683501 /loss/FRt1/detection_loss‚0.5187349888852555 0loss/FBt2/detection_loss‚0.49687128649932927 2loss/SRqqt1/assignment_loss‚0.6142076963780266 3loss/SRqqt2/assignment_loss‚0.48154558012059323 3loss/FBt1/assignment_loss‚0.0052189351618034275 1loss/SRqqt1/detection_loss‚0.7315146339920963 1loss/SRqqt2/detection_loss‚0.5441440315107906Úc.@™œ„‚(:‘ ϸíÒ—´£‹ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.293599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o8ùÍâ…‚(:× ϸíÒ窓 network.sent‚ 1.2191588e+07 network.recv‚ 1.340065e+06* disk./.usagePercent‚39.621209818695874# disk.nvme0c0n1.in‚ 5641.11328125! disk.nvme0n1.in‚ 5641.11328125 disk.sda3.out‚ 107.421875 disk.dm-1.out‚ 107.421875 disk.dm-0.in‚ 5641.11328125 disk.rbd1.out‚ 10.20703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 196.6171875 disk.drbd2165.in‚0% disk.nvme0n1.out‚7071.33447265625 disk.sda2.in‚0# memory_percent‚98.7999439239502 disk.sda.in‚0.109375 disk.sda.out‚ 107.421875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375* proc.memory.percent‚16.541028022766113$ disk./.usageGB‚590.3846588134766 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚7071.33447265625" disk.dm-0.out‚7071.33447265625# proc.memory.rssMB‚ 2710.08203125 disk.dm-1.in‚0.109375 cpu‚26.887726453915345 disk.sda1.in‚0 proc.cpu.threads‚17=øl$2†- _wandbruntime‚4259  _runtime‚4259î}dƇÀ _runtime‚ 4259.25345083 _step‚731 & lr-AdamW/pg1‚0.0007589612561099739 & lr-AdamW/pg2‚0.0007589612561099739 trainer/global_step‚9049 % _timestamp‚1.7843723178096266e+09±¶ÉÏˈŠ_runtime‚ 4259.25345083 _step‚731 & lr-AdamW/pg1‚0.0007589612561099739 & lr-AdamW/pg2‚0.0007589612561099739 trainer/global_step‚9049 % _timestamp‚1.7843723178096266e+09ÛØTáŸ2‰- _wandbruntime‚4259  _runtime‚4259•mݘD ŠÎO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85914189 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351774094516736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351774199493120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67933807 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85904212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331541325174784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326024362352640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153838017 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5340476547584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62274701 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5340388610560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41732025 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5321691042324 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340610443139072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5321604905234 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20542430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326024387256320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚,Yq2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚346433722945536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67933807 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67933805 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4259.254664421 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚346433705906176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331541522188288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85914189 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20542673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340610426099712 trainer/global_step‚9049 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62274455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚732 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚346433722945536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚346433705906176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5516962822144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85904212 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5340388610560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153847996 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5340476547584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5517134931968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351774094516736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41732028 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4259.254664421 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚346433705906176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331541522188288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85914189 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20542673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340610426099712 trainer/global_step‚9049 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62274455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚732 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚346433722945536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚346433705906176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5516962822144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85904212 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5340388610560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153847996 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5340476547584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5517134931968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351774094516736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41732028 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351774094516736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351774199493120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67933807 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85904212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331541325174784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326024362352640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153838017 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5340476547584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62274701 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5340388610560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41732025 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5321691042324 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340610443139072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5321604905234 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20542430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326024387256320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2ÜDÍJ@2Œ- _wandbruntime‚4259  _runtime‚42593sGÞ&ØM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85913493 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326059273867264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚9049 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67941026 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41736512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5322258363966 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67941026 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351811879823872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚346470833909760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351811774840320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚346470833909760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331576826265600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚345969070361804 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153864512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67941028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5517552398336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62281308 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5322172226764 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340646898135040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚4259.593957507 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚346470816870400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5340957969920 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62281080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351811879823872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5341045914112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20544571 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331577021174784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326059298770944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67941028 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85923484 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 % _timestamp‚1.7843723181499126e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5341045914112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5340957969920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345969173538366 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153854519 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993:úçˆdŽÝM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340646898135040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _runtime‚4259.593957507 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚346470816870400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5340957969920 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62281080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351811879823872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5341045914112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20544571 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331577021174784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326059298770944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67941028 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85923484 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 % _timestamp‚1.7843723181499126e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5341045914112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5340957969920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345969173538366 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153854519 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85913493 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326059273867264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_Ëe±Ðpool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚9049 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67941026 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41736512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5322258363966 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67941026 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351811879823872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚346470833909760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351811774840320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚346470833909760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331576826265600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚345969070361804 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153864512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67941028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5517552398336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62281308 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5322172226764Ý0ºÓ 2- _wandbruntime‚4259  _runtime‚4259Òv  „ 3loss/SRqqt2/assignment_loss‚0.48095388459877536 epoch‚11 trainer/global_step‚9049 $ _timestamp‚1.784372318151534e+09 _runtime‚4259.595070628 3loss/FBt2/assignment_loss‚0.0022341205074854834 /loss/FRt1/detection_loss‚0.5732586333864043 /loss/FRt2/detection_loss‚1.6121633130751083 1loss/SRqqt1/detection_loss‚0.8350817798848518 /loss/FBt1/detection_loss‚0.7608129505533725 /loss/FBt2/detection_loss‚0.4541430655391077 3loss/FBt1/assignment_loss‚8.038904972362815e-06 0loss/SRqqt2/detection_loss‚0.699512416437567 _step‚734 %loss/total_loss‚8.574549090239437 /loss/FRt1/assignment_loss‚1.669437077337382 0loss/FRt2/assignment_loss‚0.7850859674854554 2loss/SRqqt1/assignment_loss‚0.7018578425289558Ò»ˆ‘‰ %loss/total_loss‚8.574549090239437 /loss/FRt1/assignment_loss‚1.669437077337382 0loss/FRt2/assignment_loss‚0.7850859674854554 2loss/SRqqt1/assignment_loss‚0.7018578425289558 3loss/SRqqt2/assignment_loss‚0.48095388459877536 epoch‚11 trainer/global_step‚9049 $ _timestamp‚1.784372318151534e+09 _runtime‚4259.595070628 3loss/FBt2/assignment_loss‚0.0022341205074854834 /loss/FRt1/detection_loss‚0.5732586333864043 /loss/FRt2/detection_loss‚1.6121633130751083 1loss/SRqqt1/detection_loss‚0.8350817798848518 /loss/FBt1/detection_loss‚0.7608129505533725 /loss/FBt2/detection_loss‚0.4541430655391077 3loss/FBt1/assignment_loss‚8.038904972362815e-06 0loss/SRqqt2/detection_loss‚0.699512416437567 _step‚734ÞÐÇmœ’‚(:‘ Þ¸íÒ¦³Š gpu.0.gpu‚59.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚188.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.706599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U#3ñÛ“‚(:Ð Þ¸íÒ¸ÙªŒ disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 107.72265625 network.sent‚ 1.2254395e+07 disk.sda3.out‚ 107.72265625$ memory_percent‚98.80030155181885# proc.memory.rssMB‚ 2710.08203125( proc.memory.availableMB‚ 196.55859375* disk./.usagePercent‚39.623146771699936 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0" disk.dm-0.out‚7076.92041015625% disk.nvme0n1.out‚7076.92041015625 proc.cpu.threads‚17 disk.sda1.out‚0* proc.memory.percent‚16.541028022766113$ disk./.usageGB‚590.4135208129883 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.20703125 disk.nvme0c0n1.in‚5705.625' disk.nvme0c0n1.out‚7076.92041015625 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.sda.out‚ 107.72265625 disk.drbd2165.in‚0 network.recv‚ 1.344647e+06 disk.dm-0.in‚5705.625 cpu‚26.889762548573124 disk.nvme0n1.in‚5705.6256Èð—‘”‚(:† í¸íÒÏÂ׌ gpu.0.gpu‚64.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09=ÃÍ•‚(: í¸íÒéÓžŽ disk.nvme0c0n1.in‚5705.625* proc.memory.percent‚16.541028022766113 cpu‚26.891749054608745 network.sent‚ 1.2319266e+07) disk./.usagePercent‚39.62314497964719 disk.dm-1.out‚ 107.89453125 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.26953125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2710.08203125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 7079.9921875 disk.nvme0n1.in‚5705.625 disk.sda1.in‚0 disk.sda3.in‚0.109375 disk.rbd1.in‚0' proc.memory.availableMB‚ 196.4140625$ disk./.usageGB‚590.4134941101074 disk.dm-0.in‚5705.625 disk.sda.in‚0.109375$ memory_percent‚98.80118370056152 disk.sda3.out‚ 107.89453125 disk.sda1.out‚0 disk.dm-0.out‚ 7079.9921875 disk.drbd2165.in‚0 disk.sda.out‚ 107.89453125! disk.nvme0n1.out‚ 7079.9921875 network.recv‚ 1.349198e+06 disk.sda2.in‚0 disk.dm-1.in‚0.109375ÞŠ¬22–- _wandbruntime‚4281  _runtime‚4281*>žpÆ—À $ _timestamp‚1.784372340276678e+09 _runtime‚4281.720610833 _step‚735 & lr-AdamW/pg1‚0.0007579178452682642 & lr-AdamW/pg2‚0.0007579178452682642 trainer/global_step‚9099óTÎR˘Š& lr-AdamW/pg1‚0.0007579178452682642 & lr-AdamW/pg2‚0.0007579178452682642 trainer/global_step‚9099 $ _timestamp‚1.784372340276678e+09 _runtime‚4281.720610833 _step‚735ß;&!‰2™- _wandbruntime‚4281  _runtime‚4281C…—Ó'šÍO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86370114 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86380091 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327769938083840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353658247929344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327769962987520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5546476476928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784372340277822e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353658142952960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353658142952960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚4281.721823792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68294857 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5368976772608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚342434027866112 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5368976772608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154664969 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚342434044905472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68294855 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68294857 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333316414560768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333316615768576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚9099 _step‚736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5368888835584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20651996 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚347784030751054 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41956225 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0úKLØ'›ÒO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68294855 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚229 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5546652781056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784372340277822e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353658142952960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353658142952960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚4281.721823792 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68294857 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5368976772608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚342434027866112 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5368976772608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154664969 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚342434044905472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68294855 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68294857 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333316414560768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333316615768576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚9099 _step‚736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5368888835584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20651996 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚347784030751054 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41956225 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86370114 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86380091 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327769938083840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353658247929344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327769962987520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5546476476928 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚342470499901440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41960709 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154691464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353695823276544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5369546139136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 UHDeviceS¸º‘ˆ¾tatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5350656343674 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68302076 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348326382120960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348326365081600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348326382120960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚342470516940800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86389386 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5369546139136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41960712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68302076 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86379395 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353695928260096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚347821173284474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62614817 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62614594 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 _step‚737 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68302078 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86389386 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333352112610816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5547238108672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154681471 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86379395 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9099 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348326365081600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5369458194944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20654105 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327804849598464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 $ _timestamp‚1.784372340616526e+09 _runtime‚4282.060720328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146™Ü,tâ&žÜM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41960709 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154691464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353695823276544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5369546139136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5350656343674 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68302076 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348326382120960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348326365081600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348326382120960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚342470516940800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86389386 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5369546139136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41960712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68302076 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86379395 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353695928260096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚347821173284474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62614817 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62614594 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 _step‚737 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68302078 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86389386 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333352112610816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5547238108672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154681471 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86379395 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9099 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348326365081600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5369458194944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20654105 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327804849598464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 $ _timestamp‚1.784372340616526e+09 _runtime‚4282.060720328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5369458194944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚342470499901440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327á£Ô³*2Ÿ- _wandbruntime‚4282  _runtime‚4282 £t  ƒ epoch‚11 1loss/SRqqt1/detection_loss‚0.9193722568193657 1loss/SRqqt2/detection_loss‚0.6937829850448503 2loss/SRqqt1/assignment_loss‚0.6947654810821404 0loss/FBt2/assignment_loss‚0.1437514938617417 /loss/FRt1/detection_loss‚0.6183460363903405 trainer/global_step‚9099 /loss/FBt2/detection_loss‚0.6138515026059708 0loss/FRt1/assignment_loss‚1.5155329231988817 0loss/FRt2/assignment_loss‚0.7692671478702332 3loss/SRqqt2/assignment_loss‚0.49982462810532113 3loss/FBt1/assignment_loss‚0.0002495108128130688 /loss/FRt2/detection_loss‚1.5622432261639214 /loss/FBt1/detection_loss‚0.3846112017591412 % _timestamp‚1.7843723406179645e+09 $loss/total_loss‚8.41559839371472 _runtime‚4282.061893653 _step‚738Þ¼Ð<¡ˆ /loss/FBt2/detection_loss‚0.6138515026059708 0loss/FRt1/assignment_loss‚1.5155329231988817 0loss/FRt2/assignment_loss‚0.7692671478702332 3loss/SRqqt2/assignment_loss‚0.49982462810532113 3loss/FBt1/assignment_loss‚0.0002495108128130688 /loss/FRt2/detection_loss‚1.5622432261639214 /loss/FBt1/detection_loss‚0.3846112017591412 % _timestamp‚1.7843723406179645e+09 $loss/total_loss‚8.41559839371472 _runtime‚4282.061893653 _step‚738 epoch‚11 1loss/SRqqt1/detection_loss‚0.9193722568193657 1loss/SRqqt2/detection_loss‚0.6937829850448503 2loss/SRqqt1/assignment_loss‚0.6947654810821404 0loss/FBt2/assignment_loss‚0.1437514938617417 /loss/FRt1/detection_loss‚0.6183460363903405 trainer/global_step‚9099âà–-›¢‚(: ü¸íÒƒÇ÷Š gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.724000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶‹p£Õ£‚(:Ê ü¸íÒƒöŒ network.sent‚ 1.232223e+07 disk.sda2.out‚0 disk.dm-0.in‚5705.625 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚5705.625# proc.memory.rssMB‚ 2710.08203125 disk.rbd0.in‚0( proc.memory.availableMB‚ 196.66015625 proc.cpu.threads‚17 network.recv‚ 1.352533e+06$ disk./.usageGB‚590.4134864807129 disk.dm-1.out‚ 108.11328125 disk.rbd1.in‚0 disk.sda.out‚ 108.11328125 disk.nvme0n1.in‚5705.625$ disk.nvme0n1.out‚7085.6435546875! disk.dm-0.out‚7085.6435546875 disk.sda3.in‚0.109375 disk.rbd0.out‚0$ memory_percent‚98.79968166351318 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 10.26953125 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚26.89358960891353) disk./.usagePercent‚39.62314446763212 disk.dm-1.in‚0.109375& disk.nvme0c0n1.out‚7085.6435546875* proc.memory.percent‚16.541028022766113 disk.sda3.out‚ 108.11328125 disk.sda.in‚0.109375fý%2¤- _wandbruntime‚4304  _runtime‚4304k³gÇ¥Á & lr-AdamW/pg1‚0.0007568690865407553 & lr-AdamW/pg2‚0.0007568690865407553 trainer/global_step‚9149 % _timestamp‚1.7843723626906495e+09 _runtime‚4304.134443165 _step‚739íÈüÌ¦Æ & lr-AdamW/pg1‚0.0007568690865407553 & lr-AdamW/pg2‚0.0007568690865407553 trainer/global_step‚9149 % _timestamp‚1.7843723626906495e+09 _runtime‚4304.134443165 _step‚739ãndo2§- _wandbruntime‚4304  _runtime‚4304˜¦ÙÎÔ'¨ÎO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5397476997632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _runtime‚4304.135702662 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20760730 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344257646671872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86845993 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 _step‚740 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355542296365568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5397389060608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350144802328576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5397389060608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355542191389184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355542191389184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329515513815040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349636133673612 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335091671571456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843723626916876e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86836016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355542296365568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62941377 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86836016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚219 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5575967034368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346sâpT©ÓO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350144802328576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5397389060608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355542191389184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355542191389184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329515513815040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349636133673612 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335091671571456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 % _timestamp‚1.7843723626916876e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86836016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355542296365568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62941377 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86836016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚219 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5575967034368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5397476997632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 _runtime‚4304.135702662 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20760730 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344257646671872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86845993 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 _step‚740 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355542296365568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5397389060608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154äš(2ª- _wandbruntime‚4304  _runtime‚4304¥.òûÝ&«×M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚350181930332160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843723630321534e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355579976696320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155508423 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68663128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86855288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335126978950144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚741 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335127175956480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355579871712768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 _runtime‚ 4304.47621869 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20763062 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349673069854020 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68663126 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚350181913292800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5378968186180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5397958419968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42184909 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329550425329664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62947748 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚344294101667840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5398046364160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350181913292800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42184912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20762839 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5576725723136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68663128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350181930332160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5398046364160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86845297 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155518416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68663126 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329550450233344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62947974 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86855288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155518416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349673173030582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155508423§¢â&¬ÜM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335127175956480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355579871712768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 _runtime‚ 4304.47621869 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20763062 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349673069854020 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68663126 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚350181913292800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5378968186180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5397958419968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42184909 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329550425329664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62947748 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚344294101667840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5398046364160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350181913292800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42184912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20762839 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5576725723136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68663128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350181930332160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5398046364160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86845297 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155518416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68663126 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329550450233344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62947974 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86855288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155518416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349673173030582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155508423 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚350181930332160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843723630321534e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355579976696320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155508423 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68663128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86855288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335126978950144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚741åk[B 2­- _wandbruntime‚4304  _runtime‚4304¦ | ®… /loss/FRt2/detection_loss‚1.3769697584588119 1loss/SRqqt1/detection_loss‚0.8634838702458761 epoch‚11 0loss/FRt1/assignment_loss‚1.7145075446344427 /loss/FRt1/detection_loss‚0.5638858183998428 1loss/SRqqt2/detection_loss‚0.5969558650823055 %loss/total_loss‚7.573010413821114 0loss/FRt2/assignment_loss‚0.5119758194143121 2loss/SRqqt2/assignment_loss‚0.4895247693355066 0loss/FBt1/detection_loss‚0.47590465446313224 trainer/global_step‚9149 % _timestamp‚1.7843723630337305e+09 _runtime‚4304.477274752 2loss/SRqqt1/assignment_loss‚0.5392190336597196 /loss/FBt2/detection_loss‚0.4298023861774709 _step‚742 2loss/FBt1/assignment_loss‚0.002615964432185791 2loss/FBt2/assignment_loss‚0.008164929517508223ìÅ+¯Š 0loss/FRt1/assignment_loss‚1.7145075446344427 /loss/FRt1/detection_loss‚0.5638858183998428 1loss/SRqqt2/detection_loss‚0.5969558650823055 %loss/total_loss‚7.573010413821114 0loss/FRt2/assignment_loss‚0.5119758194143121 2loss/SRqqt2/assignment_loss‚0.4895247693355066 0loss/FBt1/detection_loss‚0.47590465446313224 trainer/global_step‚9149 % _timestamp‚1.7843723630337305e+09 _runtime‚4304.477274752 2loss/SRqqt1/assignment_loss‚0.5392190336597196 /loss/FBt2/detection_loss‚0.4298023861774709 _step‚742 2loss/FBt1/assignment_loss‚0.002615964432185791 2loss/FBt2/assignment_loss‚0.008164929517508223 /loss/FRt2/detection_loss‚1.3769697584588119 1loss/SRqqt1/detection_loss‚0.8634838702458761 epoch‚11æÃ0ì7œ°‚(:‘ ‹¹íÒ´³«‹ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚žëر‚(:Í ‹¹íÒŽ¤¶$ memory_percent‚98.80263805389404# proc.memory.rssMB‚ 2710.08203125 network.sent‚ 1.2385021e+07% disk.nvme0n1.out‚7237.57763671875* proc.memory.percent‚16.541028022766113 network.recv‚ 1.357036e+06# disk./.usageGB‚590.269157409668 disk.rbd1.out‚ 10.39453125 disk.nvme0c0n1.in‚5705.625 disk.rbd0.out‚0 disk.dm-0.in‚5705.625( proc.memory.availableMB‚ 196.17578125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 108.45703125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 cpu‚26.89539459079167 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 108.45703125 disk.sda.in‚0.109375 disk.nvme0n1.in‚5705.625 disk.dm-1.out‚ 108.45703125) disk./.usagePercent‚39.61345842257414 disk.sda3.in‚0.109375" disk.dm-0.out‚7237.57763671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚7237.57763671875 disk.sda1.out‚0)yƒ œ²‚(:‘ š¹íÒÙìÏ‹ gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚193.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DnUⳂ(:× š¹íÒ¶¤›Œ) disk./.usagePercent‚39.61324030415497 disk.rbd0.in‚0 disk.sda2.out‚0! disk.dm-0.out‚7247.3583984375 disk.sda.out‚ 110.53515625 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 110.53515625 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚7247.3583984375 cpu‚26.89715070898813$ disk./.usageGB‚590.2659072875977 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚98.79992008209229 disk.sda3.in‚0.109375 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.39453125" disk.nvme0c0n1.in‚ 5773.4765625 proc.cpu.threads‚17$ disk.nvme0n1.out‚7247.3583984375 disk.dm-0.in‚ 5773.4765625 disk.sda3.out‚ 110.53515625 disk.sda.in‚0.109375 disk.drbd2165.in‚0( proc.memory.availableMB‚ 196.62109375 network.sent‚ 1.2447815e+07 disk.rbd1.in‚0 network.recv‚ 1.361496e+06 disk.nvme0n1.in‚ 5773.4765625# proc.memory.rssMB‚ 2710.08203125* proc.memory.percent‚16.541028022766113D±pO2´- _wandbruntime‚4326  _runtime‚4326€8¨ÓǵÁ _runtime‚4326.478595018 _step‚743 & lr-AdamW/pg1‚0.0007558149967254472 & lr-AdamW/pg2‚0.0007558149967254472 trainer/global_step‚9199 % _timestamp‚1.7843723850347762e+09>aG»¶Æ & lr-AdamW/pg1‚0.00-–è̯07558149967254472 & lr-AdamW/pg2‚0.0007558149967254472 trainer/global_step‚9199 % _timestamp‚1.7843723850347762e+09 _runtime‚4326.478595018 _step‚743ç0Æé2·- _wandbruntime‚4326  _runtime‚4326Êq¯¾Ô'¸ÎO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20869610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42404625 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69016957 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42404628 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357426239825408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87301918 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351488030243270 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69016955 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331261089546240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352000350539776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87311895 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63274238 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5406884981448 _step‚744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5425889285632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20869383 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87311895 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352000367579136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351488133419720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346081231398912 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69016955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336866774346240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5406798844358 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357426344801792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5425977222656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚4326.479906403 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156318873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87301918 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5605659896320 % _timestamp‚1.7843723850359244e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331261114449920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87301918 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5605659896320 % _timestamp‚1.7843723850359244e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331261114449920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42404625 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69016957 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42404628 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357426239825408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87301918 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351488030243270 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69016955 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331261089546240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352000350539776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87311895 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63274238 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5406884981448 _step‚744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5425889285632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20869383 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87311895 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352000367579136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351488133419720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346081231398912 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69016955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336866774346240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5406798844358è²2¥{2º- _wandbruntime‚4326  _runtime‚4326ë©ñSÞ&»ØM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5426546589184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351525172776690 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚745 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352037478543360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _runtime‚4326.818902899 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69024178 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5407366165888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352037478543360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346117703434240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42409109 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357463920148992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87311199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336902273697280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357464025132544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69024178 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156345368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63280650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357464025132544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336902068302336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5606067241472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352037461504000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20871541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5407452303090 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843723853748028e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87321190 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87311199 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5606247732736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156335375 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20871783 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331296025964544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346117720473600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156345368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63280650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357464025132544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336902068302336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5606067241472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352037461504000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20871541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5407452303090 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843723853748028e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87321190 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87311199 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5606247732736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156335375 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20871783 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331296025964544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346117720473600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5407366165888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352037478543360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346117703434240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42409109 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357463920148992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87311199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336902273697280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357464025132544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360éIóÏa2½- _wandbruntime‚4326  _runtime‚43260£&ä ¾ƒ 1loss/FBt1/assignment_loss‚0.00239010716868604 3loss/FBt2/assignment_loss‚0.0039201935756754125 /loss/FRt2/detection_loss‚1.4778951579986217 0loss/FBt1/detection_loss‚0.37225346741615795 /loss/FBt2/detection_loss‚0.3226113809796516 /loss/FRt1/detection_loss‚0.6311185217315614 1loss/SRqqt1/detection_loss‚0.7379913896020199 _runtime‚ 4326.82013763 _step‚746 0loss/FRt2/assignment_loss‚0.6886630458757281 2loss/SRqqt2/assignment_loss‚0.5440606880298392 %loss/total_loss‚7.762456158805046 0loss/FRt1/assignment_loss‚1.6901594469964625 2loss/SRqqt1/assignment_loss‚0.6639401208649459 1loss/SRqqt2/detection_loss‚0.6274526385656957 epoch‚11 trainer/global_step‚9199 $ _timestamp‚1.784372385376599e+09]Óá·¿ˆ /loss/FRt1/detection_loss‚0.6311185217315614 1loss/SRqqt1/detection_loss‚0.7379913896020199 _runtime‚ 4326.82013763 _step‚746 0loss/FRt2/assignment_loss‚0.6886630458757281 2loss/SRqqt2/assignment_loss‚0.5440606880298392 %loss/total_loss‚7.762456158805046 0loss/FRt1/assignment_loss‚1.6901594469964625 2loss/SRqqt1/assignment_loss‚0.6639401208649459 1loss/SRqqt2/detection_loss‚0.6274526385656957 epoch‚11 trainer/global_step‚9199 $ _timestamp‚1.784372385376599e+09 1loss/FBt1/assignment_loss‚0.00239010716868604 3loss/FBt2/assignment_loss‚0.0039201935756754125 /loss/FRt2/detection_loss‚1.4778951579986217 0loss/FBt1/detection_loss‚0.37225346741615795 /loss/FBt2/detection_loss‚0.3226113809796516ê¬Ãd‘À‚(:† ©¹íÒÀÈ’ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚182.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òuFÜÁ‚(:Ñ ©¹íÒœ“ÞŽ disk.rbd0.in‚0 network.sent‚ 1.246582e+07 disk.sda1.in‚0 disk.dm-0.in‚ 5773.4765625 disk.dm-0.out‚7251.048828125& proc.memory.availableMB‚ 196.359375 network.recv‚ 1.365091e+06 disk.dm-1.out‚ 111.06640625 disk.sda.out‚ 111.06640625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 5773.4765625 disk.nvme0n1.in‚ 5773.4765625$ memory_percent‚98.80151748657227 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.rbd1.out‚ 10.45703125# proc.memory.rssMB‚ 2710.08203125 proc.cpu.threads‚17 disk.rbd0.out‚0) disk./.usagePercent‚39.61324184020018 disk.sda2.out‚0 disk.sda3.out‚ 111.06640625 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.541028022766113$ disk./.usageGB‚590.2659301757812 disk.sda3.in‚0.109375 cpu‚26.89489325524192% disk.nvme0c0n1.out‚7251.048828125# disk.nvme0n1.out‚7251.048828125}‚¸ù2Â- _wandbruntime‚4348  _runtime‚4348ºiÀôÇÃÁ & lr-AdamW/pg1‚0.0007547555927057279 & lr-AdamW/pg2‚0.0007547555927057279 trainer/global_step‚9249 % _timestamp‚1.7843724068265336e+09 _runtime‚4348.270372692 _step‚747X“€8ÌÄÆ & lr-AdamW/pg2‚0.0007547555927057279 trainer/global_step‚9249 % _timestamp‚1.7843724068265336e+09 _runtime‚4348.270372692 _step‚747 & lr-AdamW/pg1‚0.0007547555927057279ë†CÒã2Å- _wandbruntime‚4348  _runtime‚4348cíßÔ'ÆÎO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359310288261632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚359310288261632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353855898750976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338641841993216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69378005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157155804 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚9249 % _timestamp‚1.7843724068276865e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5454389510656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚214 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20978380 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69378007 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5454477447680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5634975507968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63607205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353340133165828 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353340029989378 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359310393238016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87777797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353855915790336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63607422 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87767820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20978594 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157145825 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353855898750976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5454477447680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5635151812096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5435282961156 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333006665277440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42628828 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5435196824066 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346Óñ{uÙ'ÇÓO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87777797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353855915790336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69378005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42628825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69378007 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5454389510656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _step‚748 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359310393238016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347904833165312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚347904850204672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5454389510656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚214 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20978380 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69378007 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5454477447680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5634975507968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63607205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353340133165828 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353340029989378 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359310393238016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87777797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353855915790336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63607422 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87767820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20978594 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157145825 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353855898750976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5454477447680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5635151812096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5435282961156 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333006665277440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42628828 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5435196824066 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359310288261632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚359310288261632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353855898750976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338641841993216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69378005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157155804 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 trainer/global_step‚9249 % _timestamp‚1.7843724068276865e+09 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353377069346236 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?Deviå´XË~#ceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63614082 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157162327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157172320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87777101 _step‚749 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87787092 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5454958870016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20980770 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5635568020992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69385226 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359348073568768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333041576792064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359348073568768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157162327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42633312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42633309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338677144813056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5635738026496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353377172522798 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚347941305200640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5455046814208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157172320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87777101 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69385226 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5435850282798 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353893009715200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69385228 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20980552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63613861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5435764145596 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69385228 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 trainer/global_step‚9249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202)t6ã&ÊÝM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353893009715200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69385228 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20980552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63613861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5435764145596 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69385228 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 trainer/global_step‚9249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353377069346236 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63614082 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157162327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157172320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87777101 _step‚749 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87787092 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5454958870016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20980770 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5635568020992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69385226 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359348073568768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333041576792064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359348073568768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157162327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42633312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42633309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338677144813056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5635738026496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353377172522798 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚347941305200640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5455046814208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157172320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87777101 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69385226 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5435850282798 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146ípÀ×2Ë- _wandbruntime‚4348  _runtime‚4348’è_é Ì„ 0loss/FRt1/assignment_loss‚1.6256748705425046 1loss/SRqqt2/assignment_loss‚0.506595831894083 1loss/SRqqt2/detection_loss‚0.6003059715531982 epoch‚11 0loss/FRt2/assignment_loss‚0.6386730305610164 2loss/SRqqt1/assignment_loss‚0.4833896401180413 4loss/FBt1/assignment_loss‚2.8936766902187827e-05 .loss/FRt2/detection_loss‚1.534575341286541 1loss/SRqqt1/detection_loss‚0.7186957135834177 /loss/FBt1/detection_loss‚1.0424971280735917 %loss/total_loss‚8.171291361842773 /loss/FRt1/detection_loss‚0.5772639058677393 /loss/FBt2/detection_loss‚0.4393475096686884 % _timestamp‚1.7843724071774943e+09 _runtime‚4348.621026016 _step‚750 2loss/FBt2/assignment_loss‚0.004243481927050418 trainer/global_step‚9249òéKµÍ‰ 0loss/FRt1/assignment_loss‚1.6256748705425046 1loss/SRqqt2/assignment_loss‚0.506595831894083 1loss/SRqqt2/detection_loss‚0.6003059715531982 epoch‚11 0loss/FRt2/assignment_loss‚0.6386730305610164 2loss/SRqqt1/assignment_loss‚0.4833896401180413 4loss/FBt1/assignment_loss‚2.8936766902187827e-05 .loss/FRt2/detection_loss‚1.534575341286541 1loss/SRqqt1/detection_loss‚0.7186957135834177 /loss/FBt1/detection_loss‚1.0424971280735917 %loss/total_loss‚8.171291361842773 /loss/FRt1/detection_loss‚0.5772639058677393 /loss/FBt2/detection_loss‚0.4393475096686884 % _timestamp‚1.7843724071774943e+09 _runtime‚4348.621026016 _step‚750 2loss/FBt2/assignment_loss‚0.004243481927050418 trainer/global_step‚9249î&’œÎ‚(:„ ¸¹íÒð¶àŠ gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚187.18' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p(ŽçÏ‚(:Ü ¸¹íÒѽ¬Œ disk.rbd0.out‚0' proc.memory.availableMB‚ 213.2265625* proc.memory.percent‚15.936517715454102 disk.sda3.out‚ 112.75390625 network.recv‚ 1.369433e+06 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 112.75390625 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.45703125" proc.memory.rssMB‚ 2611.0390625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-0.in‚ 5773.48828125 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0% disk.nvme0n1.out‚7262.07373046875$ memory_percent‚98.69856834411621 cpu‚26.89265699207782$ disk./.usageGB‚590.2674751281738 disk.sda1.out‚0 disk.sda.out‚ 112.75390625! disk.nvme0n1.in‚ 5773.48828125 disk.rbd0.in‚0* disk./.usagePercent‚39.613345523251546 disk.sda3.in‚0.109375# disk.nvme0c0n1.in‚ 5773.48828125' disk.nvme0c0n1.out‚7262.07373046875 network.sent‚ 1.2515647e+07" disk.dm-0.out‚7262.07373046875 disk.rbd1.in‚0+ä^›Ð‚(: ǹíÒˆø´Š gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚178.858' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.77160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FüpeæÑ‚(:Û ǹíÒ¨•¤Œ disk.sda3.in‚0.109375 disk.drbd2165.in‚0% proc.memory.availableMB‚ 212.96875' disk.nvme0c0n1.out‚7296.82763671875! disk.nvme0n1.in‚ 5773.51171875 disk.rbd0.out‚0" disk.dm-0.out‚7296.82763671875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 115.30078125" proc.memory.rssMB‚ 2611.0390625% disk.nvme0n1.out‚7296.82763671875 disk.rbd0.in‚0 disk.sda.out‚ 115.30078125 disk.rbd1.out‚ 10.45703125 disk.sda.in‚0.109375# disk.nvme0c0n1.in‚ 5773.51171875 disk.sda1.in‚0 disk.dm-1.in‚0.109375 disk.drbd2165.out‚0$ memory_percent‚98.70014190673828* proc.memory.percent‚15.936517715454102* disk./.usagePercent‚39.624648767903324$ disk./.usageGB‚590.4359016418457 disk.rbd1.in‚0 disk.dm-1.out‚ 115.30078125 disk.dm-0.in‚ 5773.51171875 cpu‚26.890520141688285 network.sent‚ 1.2578187e+07 network.recv‚ 1.373545e+06 disk.sda1.out‚0 disk.sda2.in‚0 Ñ4¢2Ò- _wandbruntime‚4370  _runtime‚4370„Ï£ÆÓÀ & lr-AdamW/pg2‚0.0007536908914501028 trainer/global_step‚9299 $ _timestamp‚1.784372428723927e+09 _runtime‚4370.167715894 _step‚751 & lr-AdamW/pg1‚0.0007536908914501028EEüËÔÅ _runtime‚4370.167715894 _step‚751 & lr-AdamW/pg1‚0.0007536908914501028 & lr-AdamW/pg2‚0.0007536908914501028 trainer/global_step‚9299 $ _timestamp‚1.784372428723927e+09ï[^¸2Õ- _wandbruntime‚4370  _runtime‚4370vžù;"ÖÎO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88233722 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5463680940864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚355192029735486 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5664699780608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21087387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340416760290816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 % _timestamp‚1.7843724287249677e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361194336697856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42853028 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5482977672704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4370.168928043 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355711464001536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355711464001536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5482977672704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5482889735680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340416965692928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334752241008640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5664519282176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69739057 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69739055 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355711446962176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69739057 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355711446962176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5482889735680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361194336697856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚355192132911936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42853025 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 trainer/global_step‚9299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157982756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157972777 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88233722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88243699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21087622 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88243699 N@D¹(€àÎeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚349728434931712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361194441674240 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69739055 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349728451971072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607dy»Ù'×ÓO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355711446962176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5482889735680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361194336697856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚355192132911936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42853025 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 trainer/global_step‚9299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157982756 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157972777 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88233722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88243699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21087622 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88243699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚349728434931712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361194441674240 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69739055 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349728451971072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88233722 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5463680940864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚355192029735486 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5664699780608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21087387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340416760290816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 % _timestamp‚1.7843724287249677e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361194336697856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42853028 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5482977672704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4370.168928043 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355711464001536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355711464001536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5482977672704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5482889735680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340416965692928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334752241008640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5664519282176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69739057 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69739055 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355711446962176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69739057 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ðÙäÆ2Ø- _wandbruntime‚4370  _runtime‚4370¡#ÍÞ&ÙØM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340452261499904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _runtime‚4370.505365327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355748557926400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69746278 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355229172268906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157989279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚753 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361232122004992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21089842 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21089621 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5483547039232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42857512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355748574965760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69746278 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69746276 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355229069092344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361232122004992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69746276 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334787152523264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88243003 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334787177426944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349764906967040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5464248262506 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157989279 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63947130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63947354 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157999272 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361232017021440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5483459095040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42857509 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154t@žã&ÚÝM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334787152523264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88243003 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334787177426944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349764906967040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5464248262506 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157989279 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63947130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63947354 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157999272 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361232017021440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5483459095040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42857509 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340452261499904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _runtime‚4370.505365327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355748557926400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69746278 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355229172268906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157989279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚753 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚361232122004992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21089842 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21089621 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5483547039232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42857512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355748574965760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69746278 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69746276 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355229069092344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361232122004992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69746276 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ñ­“‹Œ2Û- _wandbruntime‚4370  _runtime‚4370 ¬ò Ü„ _runtime‚4370.506425619 .loss/FRt2/detection_loss‚1.542799857672213 /loss/FBt2/detection_loss‚0.4302952383631762 epoch‚11 trainer/global_step‚9299 /loss/FRt1/assignment_loss‚1.695609524611413 2loss/SRqqt2/assignment_loss‚0.4804442914295976 /loss/FRt1/detection_loss‚0.5947402700805082 1loss/SRqqt2/detection_loss‚0.6618156399172813 0loss/FRt2/assignment_loss‚0.8173413654653038 2loss/SRqqt1/assignment_loss‚0.6009951521342384 2loss/FBt2/assignment_loss‚0.002219186780485081 %loss/total_loss‚8.211476071771203 % _timestamp‚1.7843724290628803e+09 _step‚754 4loss/FBt1/assignment_loss‚0.00042144309323403445 1loss/SRqqt1/detection_loss‚0.8143588364073157 /loss/FBt1/detection_loss‚0.57043526581643779ù©ò݉ /loss/FBt1/detection_loss‚0.5704352658164377 _runtime‚4370.506425619 .loss/FRt2/detection_loss‚1.542799857672213 /loss/FBt2/detection_loss‚0.4302952383631762 epoch‚11 trainer/global_step‚9299 /loss/FRt1/assrz݆ignment_loss‚1.695609524611413 2loss/SRqqt2/assignment_loss‚0.4804442914295976 /loss/FRt1/detection_loss‚0.5947402700805082 1loss/SRqqt2/detection_loss‚0.6618156399172813 0loss/FRt2/assignment_loss‚0.8173413654653038 2loss/SRqqt1/assignment_loss‚0.6009951521342384 2loss/FBt2/assignment_loss‚0.002219186780485081 %loss/total_loss‚8.211476071771203 % _timestamp‚1.7843724290628803e+09 _step‚754 4loss/FBt1/assignment_loss‚0.00042144309323403445 1loss/SRqqt1/detection_loss‚0.8143588364073157ò`ñ†4œÞ‚(:‘ Ö¹íÒ°«¬Œ gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚153.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.631200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã'¸âß‚(:× Ö¹íÒã•åŽ network.recv‚ 1.376867e+06" disk.nvme0c0n1.in‚ 5806.8515625 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚7308.4619140625 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚7308.4619140625 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚98.70057106018066* disk./.usagePercent‚39.635362171196206 proc.cpu.threads‚17$ disk./.usageGB‚590.5955390930176 disk.sda2.in‚0 disk.dm-1.in‚0.109375 network.sent‚ 1.2583152e+07 disk.sda3.out‚ 116.83984375 disk.nvme0n1.in‚ 5806.8515625 disk.rbd0.out‚0 disk.dm-1.out‚ 116.83984375" proc.memory.rssMB‚ 2611.0390625 disk.rbd1.out‚ 10.51953125& disk.nvme0c0n1.out‚7308.4619140625' proc.memory.availableMB‚ 212.8984375* proc.memory.percent‚15.936517715454102 cpu‚26.888308527909636 disk.sda.out‚ 116.83984375 disk.drbd2165.out‚0 disk.dm-0.in‚ 5806.8515625ïÁhJ2à- _wandbruntime‚4391  _runtime‚4391z·ÇáÁ & lr-AdamW/pg1‚0.0007526209100119235 & lr-AdamW/pg2‚0.0007526209100119235 trainer/global_step‚9349 % _timestamp‚1.7843724504779952e+09 _runtime‚4391.921823705 _step‚755AX(ƒÌâÆ _step‚755 & lr-AdamW/pg1‚0.0007526209100119235 & lr-AdamW/pg2‚0.0007526209100119235 trainer/global_step‚9349 % _timestamp‚1.7843724504779952e+09 _runtime‚4391.921823705ó›¶ñ2ã- _wandbruntime‚4391  _runtime‚4391¸_‚…Ô'äÎO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88699624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70100107 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363078385134080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363078385134080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357567012212736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357566995173376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64274671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336497816739840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5694185262080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88709601 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70100105 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚342191829892096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158809708 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843724504790382e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21197443 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5511389960704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363078490110464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5511477897728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64274442 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 _runtime‚4391.926582638 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357566995173376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5492078920572 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357044029481594 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43077225 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357567012212736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70100107 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342192026905600®=è Ù'åÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351552053737472 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43077228 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351552036698112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5511477897728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843724504790382e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21197443 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5511389960704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363078490110464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5511477897728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64274442 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 _runtime‚4391.926582638 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357566995173376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5492078920572 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357044029481594 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43077225 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357567012212736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70100107 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342192026905600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88699624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70100107 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363078385134080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363078385134080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357567012212736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357566995173376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64274671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336497816739840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚226 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5694185262080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88709601 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70100105 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚342191829892096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158809708 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70107326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43081712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363116170441216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5694775138304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88718896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336532753158144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21199386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5511959320064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5492560105012 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357604106137600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64281327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5512047264256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158826224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43081709 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351588508733440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342227528296448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357604106137600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336532728254464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357081068838452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5492646242214 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363116065457664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158816231 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚357604123176960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70107328 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70107328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64281095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88718896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚757 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5512047264256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5511959320064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚342227327095808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351588525772800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70107326 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88708905 LCDeviceStatsMonitor.on_train_batch_end/allocÊ58qœation.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357604123176960 _runtime‚4392.263056975Õ1:£ã&èÝM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64281327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5512047264256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158826224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43081709 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351588508733440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342227528296448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357604106137600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336532728254464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357081068838452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5492646242214 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363116065457664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158816231 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚357604123176960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70107328 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70107328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64281095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88718896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 _step‚757 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5512047264256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5511959320064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚342227327095808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351588525772800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70107326 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88708905 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357604123176960 _runtime‚4392.263056975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70107326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43081712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363116170441216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5694775138304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88718896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336532753158144 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21199386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5511959320064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5492560105012 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357604106137600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0õ’ŸwR2é- _wandbruntime‚4392  _runtime‚4392ï¢$ ê„ /loss/FBt2/detection_loss‚0.6365652956084301 % _timestamp‚1.7843724508209999e+09 0loss/FRt2/assignment_loss‚0.7465379718710615 2loss/SRqqt2/assignment_loss‚0.5418120823549523 4loss/FBt2/assignment_loss‚0.00034030101630941475 1loss/SRqqt1/detection_loss‚0.8247799413359683 _step‚758 /loss/FRt1/detection_loss‚0.5787882898427898 epoch‚11 $loss/total_loss‚8.60713374957786 2loss/SRqqt1/assignment_loss‚0.6374230866868342 2loss/FBt1/assignment_loss‚0.016515843229497246 1loss/SRqqt2/detection_loss‚0.6873213404687283 .loss/FBt1/detection_loss‚0.744296911599425 trainer/global_step‚9349 _runtime‚4392.264551527 0loss/FRt1/assignment_loss‚1.5000596350763216 /loss/FRt2/detection_loss‚1.6926930504875424很ë‰ 0loss/FRt1/assignment_loss‚1.5000596350763216 /loss/FRt2/detection_loss‚1.6926930504875424 /loss/FBt2/detection_loss‚0.6365652956084301 % _timestamp‚1.7843724508209999e+09 0loss/FRt2/assignment_loss‚0.7465379718710615 2loss/SRqqt2/assignment_loss‚0.5418120823549523 4loss/FBt2/assignment_loss‚0.00034030101630941475 1loss/SRqqt1/detection_loss‚0.8247799413359683 _step‚758 /loss/FRt1/detection_loss‚0.5787882898427898 epoch‚11 $loss/total_loss‚8.60713374957786 2loss/SRqqt1/assignment_loss‚0.6374230866868342 2loss/FBt1/assignment_loss‚0.016515843229497246 1loss/SRqqt2/detection_loss‚0.6873213404687283 .loss/FBt1/detection_loss‚0.744296911599425 trainer/global_step‚9349 _runtime‚4392.264551527ö[P•‘ì‚(:† å¹íÒÜäœ gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚182.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—b‰íÒí‚(:Ç å¹íÒ˜ü¢Ž disk.dm-0.out‚7641.830078125 disk.sda3.out‚ 119.01171875 disk.drbd2165.in‚0$ memory_percent‚98.69880676269531 cpu‚26.88603708420533 network.recv‚ 1.380555e+06$ disk./.usageGB‚590.8406677246094* proc.memory.percent‚15.936517715454102 disk.dm-1.out‚ 119.01171875 disk.rbd1.out‚ 10.51953125 network.sent‚ 1.2645107e+07* disk./.usagePercent‚39.651812959338955 disk.dm-0.in‚ 5861.1875 disk.sda3.in‚0.109375 disk.nvme0c0n1.in‚ 5861.1875 disk.sda1.out‚0$ proc.memory.availableMB‚213.1875" proc.memory.rssMB‚ 2611.0390625# disk.nvme0n1.out‚7641.830078125 disk.rbd0.out‚0 disk.sda.in‚0.109375% disk.nvme0c0n1.out‚7641.830078125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 119.01171875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 5861.1875 disk.sda2.out‚0¶Ó†$œî‚(:‘ ô¹íÒ·¤£Š gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.111600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ“žÐï‚(:Å ô¹íÒ¤èìŒ) disk./.usagePercent‚39.83079064260922 disk.sda3.in‚0.109375 disk.sda1.out‚0 network.recv‚ 1.384846e+06 disk.dm-1.in‚0.109375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 5861.1875' disk.nvme0c0n1.out‚7708.16943359375 disk.rbd0.out‚0$ memory_percent‚98.70083332061768" proc.memory.rssMB‚ 2611.0390625 disk.dm-1.out‚120.9375 disk.sda1.in‚0 cpu‚26.884017782687206 disk.nvme0n1.in‚ 5861.1875 disk.rbd0.in‚0 disk.sda.out‚120.9375$ disk./.usageGB‚593.5075645446777 disk.sda.in‚0.109375 disk.sda2.out‚0 disk.dm-0.in‚ 5861.1875 disk.sda3.out‚120.9375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.58203125% disk.nvme0n1.out‚7708.16943359375* proc.memory.percent‚15.936517715454102 network.sent‚ 1.2708036e+07 disk.sda2.in‚0" disk.dm-0.out‚7708.16943359375( proc.memory.availableMB‚ 212.85546875 proc.cpu.threads‚17‰¡ª{2ð- _wandbruntime‚4413  _runtime‚4413K•’7ÆñÀ _runtime‚4413.824551445 _step‚759 & lr-AdamW/pg1‚0.0007515456655291148 & lr-AdamW/pg2‚0.0007515456655291148 trainer/global_step‚9399 $ _timestamp‚1.784372472379201e+09¬ùLËòÅ _step‚759 & lr-AdamW/pg1‚0.0007515456655291148 & lr-AdamW/pg2‚0.0007515456655291148 trainer/global_step‚9399 $ _timestamp‚1.784372472379201e+09 _runtime‚4413.824551445÷ýÖ312ó- _wandbruntime‚4413  _runtime‚4413œôÔ'ôÎO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70461157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338243417374720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843724723799503e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5520476900280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64607357 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5723477231104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚760 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89175503 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43301428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338243392471040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159626681 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364962433570304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359422560423936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89175503 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5539978122752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70461157 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70461155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚364962538546688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89165526 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343966869702144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚4413.824659458 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21306163 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364962538546688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358896029227702 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚359422560423936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353375638464512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚364962433570304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚359422543384576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43301425 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64607591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5539978122752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21305932 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358896132404152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70461155 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5539890185728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5723655632384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353375655503872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343967073007104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89165526 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159636660 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 trainer/global_step‚9399 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359422543384576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 °žY5õÓO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚359422543384576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43301425 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64607591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5539978122752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21305932 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358896132404152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70461155 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5539890185728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5723655632384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353375655503872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343967073007104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89165526 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159636660 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 trainer/global_step‚9399 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359422543384576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70461157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338243417374720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843724723799503e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5520476900280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64607357 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5723477231104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚760 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89175503 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43301428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338243392471040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_trã‘ܤ ain_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159626681 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364962433570304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359422560423936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89175503 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5539978122752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70461157 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70461155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚364962538546688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89165526 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343966869702144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚4413.824659458 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21306163 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364962538546688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358896029227702 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚359422560423936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353375638464512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚364962433570304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146øQ|¸‘2ö- _wandbruntime‚4414  _runtime‚4414b@¾fÞ&÷ØM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159653176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70468376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89184798 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64614032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5540547489280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21308123 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5540459545088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359459671388160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353412110499840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70468378 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5724238627328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159643183 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89174807 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70468378 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89184798 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚761 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5540547489280 trainer/global_step‚9399 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843724727168818e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159653176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3^G¡ã&øÝM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5724068621824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43305909 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344002567516672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353412127539200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚761 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5540547489280 trainer/global_step‚9399 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843724727168818e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159653176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159653176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70468376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89184798 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64614032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5540547489280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21308123 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5540459545088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359459671388160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353412110499840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70468378 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5724238627328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159643183 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89174807 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70468378 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89184798 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366846482006528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70822207 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21414809 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360748132150260 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64940664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361278091595776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366846482006528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89631428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70822205 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5568390410752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89641405 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 trainer/global_step‚9449 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89631428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366846586982912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64940434 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361278108635136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43525625 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361278091595776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366846482006528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89631428 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70822205 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5568390410752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89641405 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 trainer/global_step‚9449 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89631428 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366846586982912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64940434 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361278108635136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43525625 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366846482006528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70822207ü4¹‡Ð2„- _wandbruntime‚4435  _runtime‚4435¬ ÇÞ&…ØM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89640709 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5569047714304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340023904620544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366884267313664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70829426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355235712266240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5568959770112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345777422097920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _step‚765 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5549356064428 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5569047714304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70829428 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340023879716864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753542381056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355235729305600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843724945298624e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89640709 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43530112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89650700 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361315202560000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5549442201630 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366884267313664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366884162330112 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64947108 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160480128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43530109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21416999 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345777623298560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 trainer/global_step‚9449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360785171507230 _runtime‚4435.973922082 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160480128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70829426 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160470135 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361315219599360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360785068330668ù_sp"†ÝM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70829426 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160470135 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361315219599360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360785068330668 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89640709 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5569047714304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340023904620544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366884267313664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70829426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355235712266240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5568959770112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345777422097920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _step‚765 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5549356064428 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5569047714304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70829428 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340023879716864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753542381056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355235729305600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843724945298624e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89640709 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43530112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89650700 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361315202560000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5549442201630 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366884267313664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366884162330112 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64947108 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 L>DeviceStat\¼ssMonitor.on_train_batch_end/allocation.all.allocated‚ 160480128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43530109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21416999 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345777623298560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 trainer/global_step‚9449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360785171507230 _runtime‚4435.973922082 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160480128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680ý@Κ2‡- _wandbruntime‚4435  _runtime‚4435vRh– ˆ„ 1loss/SRqqt1/detection_loss‚0.8395466498628302 0loss/FBt2/detection_loss‚0.43556294253018374 %loss/total_loss‚8.460542546710148 _step‚766 2loss/SRqqt1/assignment_loss‚0.6755709765042109 2loss/SRqqt2/assignment_loss‚0.5737447912707831 _runtime‚4435.975529427 0loss/FRt1/assignment_loss‚1.7088989776084857 1loss/FBt2/assignment_loss‚0.01819622686199634 /loss/FRt2/detection_loss‚1.5901933379986501 1loss/SRqqt2/detection_loss‚0.6657178749858289 0loss/FRt2/assignment_loss‚0.6370008980974238 /loss/FRt1/detection_loss‚0.5776104530207511 /loss/FBt1/detection_loss‚0.7286186548457905 epoch‚11 trainer/global_step‚9449 % _timestamp‚1.7843724945319839e+09 2loss/FBt1/assignment_loss‚0.009880763123212753ükà‰‰ 1loss/SRqqt2/detection_loss‚0.6657178749858289 0loss/FRt2/assignment_loss‚0.6370008980974238 /loss/FRt1/detection_loss‚0.5776104530207511 /loss/FBt1/detection_loss‚0.7286186548457905 epoch‚11 trainer/global_step‚9449 % _timestamp‚1.7843724945319839e+09 2loss/FBt1/assignment_loss‚0.009880763123212753 1loss/SRqqt1/detection_loss‚0.8395466498628302 0loss/FBt2/detection_loss‚0.43556294253018374 %loss/total_loss‚8.460542546710148 _step‚766 2loss/SRqqt1/assignment_loss‚0.6755709765042109 2loss/SRqqt2/assignment_loss‚0.5737447912707831 _runtime‚4435.975529427 0loss/FRt1/assignment_loss‚1.7088989776084857 1loss/FBt2/assignment_loss‚0.01819622686199634 /loss/FRt2/detection_loss‚1.5901933379986501þ‹ÕÛ\›Š‚(: ’ºíÒ‘¨òŠ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚183.984' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.796800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ€‘ñç‹‚(:Ü ’ºíÒŠ¦„Ž network.sent‚ 1.2788667e+07 disk.dm-1.in‚0.109375# disk.dm-0.out‚10314.01123046875" proc.memory.rssMB‚ 2611.0390625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚10314.01123046875 disk.sda1.out‚0$ disk./.usageGB‚593.4615669250488! disk.nvme0n1.in‚ 5861.20703125 disk.rbd0.out‚0 disk.sda.in‚0.109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 127.09765625 disk.sda.out‚ 127.09765625# disk.nvme0c0n1.in‚ 5861.20703125 proc.cpu.threads‚17 disk.dm-1.out‚ 127.09765625 network.recv‚ 1.392917e+06 disk.rbd1.out‚ 10.64453125* proc.memory.percent‚15.936517715454102% proc.memory.availableMB‚ 212.84375) disk./.usagePercent‚39.82770370376135& disk.nvme0n1.out‚10314.01123046875 disk.sda1.in‚0 cpu‚26.879633818661592 disk.dm-0.in‚ 5861.20703125 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0.109375# memory_percent‚98.7009048461914Y‚¯GœŒ‚(:‘ ¡ºíÒžÓçŠ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ‹ô®è‚(:Ý ¡ºíÒ¡ˆ’! disk.nvme0n1.in‚ 5948.35546875# disk.dm-0.out‚10607.98974609375 network.sent‚ 1.2838489e+07 network.recv‚ 1.397157e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2611.0390625 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 127.61328125 disk.sda3.out‚ 127.61328125$ memory_percent‚98.70054721832275( proc.memory.availableMB‚ 212.90234375 disk.dm-0.in‚ 5948.35546875 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.64453125& disk.nvme0n1.out‚10607.98974609375 proc.cpu.threads‚17 disk.sda.out‚ 127.61328125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 5948.35546875 disk.sda2.out‚0 disk.sda.in‚0.109375( disk./.usagePercent‚39.8197390533636* proc.memory.percent‚15.936517715454102 cpu‚26.87753790965077 disk.sda2.in‚0( disk.nvme0c0n1.out‚10607.98974609375# disk./.usageGB‚593.342887878418 disk.sda1.out‚0 disk.sda3.in‚0.109375­902Ž- _wandbruntime‚4457  _runtime‚4457!jcâÇÁ trainer/global_step‚9499 % _timestamp‚1.7843725162135239e+09 _runtime‚4457.657331432 _step‚767 & lr-AdamW/pg1‚0.0007493794564025231 & lr-AdamW/pg2‚0.0007493794564025231©U0ÌÆ _step‚767 & lr-AdamW/pg1‚0.0007493794564025231 & lr-AdamW/pg2‚0.0007493794564025231 trainer/global_step‚9499 % _timestamp‚1.7843725162135239e+09 _runtime‚4457.657331432ÿ5I42‘- _wandbruntime‚4457  _runtime‚4457ß/ƒÔ'’ÎO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363133639806976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161280585 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363133639806976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43749828 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65273186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43749825 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 % _timestamp‚1.7843725162146962e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368730530442752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71183257 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341734543933440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5596890635776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341734568837120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368730530442752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161290564 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363133656846336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71183255 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90107307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 _step‚768 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21523581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71183255 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90097330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368730635419136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347517190712320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5577272859696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5577186722606 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65273409 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71183257 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5596978572800ƒ£ŸÙ'“ÓO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368730635419136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347516993698816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90097330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90107307 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5782621875200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363133656846336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362600028719918 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161280585 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363133639806976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43749828 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65273186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43749825 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 % _timestamp‚1.7843725162146962e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368730530442752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71183257 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚341734543933440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5596890635776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚223 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚341734568837120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368730530442752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161290564 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363133656846336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71183255 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90107307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 _step‚768 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21523581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71183255 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90097330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368730635419136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347517190712320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5577272859696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5577186722606 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65273409 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71183257 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5596978572800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363133639806976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Jö•ØM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5783042782208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5577754044136 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65279861 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357059314032640 % _timestamp‚1.7843725165548813e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90106611 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161307080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347552688945152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71190476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚y““nè368768210766336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5783208593408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚4457.998917537 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341769480351744 trainer/global_step‚9499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368768315749888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5597459995136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43754312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43754309 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341769455448064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71190478 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21525552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362637171253338 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161297087 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368768315749888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5597459995136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363170750771200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161307080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347552688945152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190714880 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71190476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368768210766336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5783208593408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚4457.998917537 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341769480351744 trainer/global_step‚9499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368768315749888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5597459995136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43754312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43754309 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341769455448064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71190478 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21525552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362637171253338 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161297087 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368768315749888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5597459995136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363170750771200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65279861 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357059314032640 % _timestamp‚1.7843725165548813e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90106611 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ݧ†2—- _wandbruntime‚4457  _runtime‚4457¹Û ˜ 0loss/SRqqt2/detection_loss‚0.711596943169119 /loss/FBt1/detection_loss‚0.5857505874538963 /loss/FBt2/detection_loss‚0.4498693650503727 .loss/FRt1/assignment_loss‚1.53782351902553 2loss/SRqqt1/assignment_loss‚0.5462165386978565 3loss/FBt2/assignment_loss‚0.0002104697710998819 epoch‚11 trainer/global_step‚9499 % _timestamp‚1.7843725165563967e+09 3loss/FBt1/assignment_loss‚6.838334074029976e-05 .loss/FRt2/detection_loss‚1.597813089589181 1loss/SRqqt1/detection_loss‚0.7317329343481872 _runtime‚4457.999933765 _step‚770 /loss/FRt2/assignment_loss‚0.660231415492793 2loss/SRqqt2/assignment_loss‚0.6460713235567629 /loss/FRt1/detection_loss‚0.5822016364841589 %loss/total_loss‚8.049586205979697*‡{Q ™† 3loss/FBt1/assignment_loss‚6.838334074029976e-05 .loss/FRt2/detection_loss‚1.597813089589181 1loss/SRqqt1/detection_loss‚0.7317329343481872 _runtime‚4457.999933765 _step‚770 /loss/FRt2/assignment_loss‚0.660231415492793 2loss/SRqqt2/assignment_loss‚0.6460713235567629 /loss/FRt1/detection_loss‚0.5822016364841589 %loss/total_loss‚8.049586205979697 0loss/SRqqt2/detection_loss‚0.711596943169119 /loss/FBt1/detection_loss‚0.5857505874538963 /loss/FBt2/detection_loss‚0.4498693650503727 .loss/FRt1/assignment_loss‚1.53782351902553 2loss/SRqqt1/assignment_loss‚0.5462165386978565 3loss/FBt2/assignment_loss‚0.0002104697710998819 epoch‚11 trainer/global_step‚9499 % _timestamp‚1.7843725165563967e+09‚j×èåÜš‚(:Ñ °ºíÒ$ disk./.usageGB‚593.3405723571777 disk.dm-1.out‚ 131.6015625" proc.memory.rssMB‚ 2611.0390625* proc.memory.percent‚15.936517715454102 cpu‚26.875441432899848 disk.dm-0.in‚ 5948.38671875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 131.6015625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0) disk./.usagePercent‚39.81958365679032 disk.dm-0.out‚ 10634.4296875 disk.drbd2165.in‚0 disk.rbd1.out‚ 10.70703125! disk.nvme0n1.in‚ 5948.38671875 disk.sda1.out‚0 network.sent‚ 1.2843398e+07 network.recv‚ 1.400348e+06 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.rbd0.out‚0( proc.memory.availableMB‚ 213.45703125 disk.sda2.in‚0 disk.sda.in‚0.109375 disk.sda.out‚ 131.6015625# disk.nvme0c0n1.in‚ 5948.38671875$ disk.nvme0c0n1.out‚ 10634.4296875$ memory_percent‚98.69716167449951 disk.drbd2165.out‚0" disk.nvme0n1.out‚ 10634.4296875]E>¶œ›‚(:‘ °ºíÒح׊ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¸ÊE2œ- _wandbruntime‚4479  _runtime‚4479°˜ÇÁ trainer/global_step‚9549 % _timestamp‚1.7843725379767866e+09 _runtime‚4479.420622843 _step‚771 & lr-AdamW/pg1‚0.0007482885264549758 & lr-AdamW/pg2‚0.0007482885264549758ì§=ÌžÆ _runtime‚4479.420622843 _step‚771 & lr-AdamW/pg1‚0.0007482885264549758 & lr-AdamW/pg2‚0.0007482885264549758 trainer/global_step‚9549 % _timestamp‚1.7843725379767866e+09ƒ\ÏS2Ÿ- _wandbruntime‚4479  _runtime‚4479šdË{Ô' ÎO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370614578878976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5605584702314 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5812149455872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364989188018176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364989188018176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90563232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343480144568320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5811973151744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349292092816384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370614578878976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90563232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71544307 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90573209 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43974028 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚364452028466026 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358846460803072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370614683855360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚364452131642476 trainer/global_step‚9549 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71544305 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21632425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843725379778748e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162117516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5605670839404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370614683855360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65606450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _step‚772 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71544307 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5625478797824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43974025 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349292294024192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343480119664640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358846443763712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364989188018176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364989188018176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90563232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343480144568320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5811973151744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349292092816384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370614578878976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90563232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71544307 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90573209 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43974028 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚364452028466026 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358846460803072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 GADeviceStatsMonitor.on_train_batch_start/active.«áî‡large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370614683855360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚364452131642476 trainer/global_step‚9549 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71544305 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21632425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843725379778748e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162117516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5605670839404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370614683855360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65606450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _step‚772 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71544307 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5625478797824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346„Y®2¢- _wandbruntime‚4479  _runtime‚4479~¶hOÞ&£ØM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5625960220160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚364489170999446 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71551526 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370652364186112 % _timestamp‚1.7843725383166714e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5606152023844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5812568011776 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162124039 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5626048164352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370652364186112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162134032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365026298982400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343515031179264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4479.760759615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90572513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚219 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365026298982400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162124039 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349327599191040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365026316021760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71551526 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21634802 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90582504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65613314 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 _step‚773 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65613095 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5812735920128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71551528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5625960220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚358882915799040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚364489067822884 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343515056082944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365026316021760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21634586!¶Òã&¤ÝM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21634802 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90582504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65613314 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 _step‚773 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65613095 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5812735920128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71551528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5625960220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚358882915799040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚364489067822884 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343515056082944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365026316021760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21634586 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5625960220160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚364489170999446 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71551526 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370652364186112 % _timestamp‚1.7843725383166714e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5606152023844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5812568011776 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162124039 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5626048164352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370652364186112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162134032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365026298982400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343515031179264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4479.760759615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90572513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚219 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365026298982400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162124039 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349327599191040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365026316021760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71551526 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473…ã˜Ä2¥- _wandbruntime‚4479  _runtime‚4479 … ¦ƒ $ _timestamp‚1.784372538318515e+09 _step‚774 0loss/FRt1/assignment_loss‚1.6159347550393979 2loss/SRqqt1/assignment_loss‚0.6280656063854942 /loss/FRt1/detection_loss‚0.5345304911365668 /loss/FRt2/detection_loss‚1.4750628456713097 1loss/SRqqt1/detection_loss‚0.6868584831546554 %loss/total_loss‚8.251168458991824 epoch‚11 2loss/SRqqt2/assignment_loss‚0.5276496838923944 3loss/FBt1/assignment_loss‚4.932983725588521e-05 1loss/FBt2/assignment_loss‚0.01935970969913616 /loss/FBt1/detection_loss‚0.7690103611460438 _runtime‚4479.762054259 0loss/FRt2/assignment_loss‚0.6991375404419286 1loss/SRqqt2/detection_loss‚0.6566855124550717 /loss/FBt2/detection_loss‚0.6388241401325692 trainer/global_step‚9549ôäS¦§ˆ trainer/global_step‚9549 $ _timestamp‚1.784372538318515e+09 _step‚774 0loss/FRt1/assignment_loss‚1.6159347550393979 2loss/SRqqt1/assignment_loss‚0.6280656063854942 /loss/FRt1/detection_loss‚0.5345304911365668 /loss/FRt2/detection_loss‚1.4750628456713097 1loss/SRqqt1/detection_loss‚0.6868584831546554 %loss/total_loss‚8.251168458991824 epoch‚11 2loss/SRqqt2/assignment_loss‚0.5276496838923944 3loss/FBt1/assignment_loss‚4.932983725588521e-05 1loss/FBt2/assignment_loss‚0.01935970969913616 /loss/FBt1/detection_loss‚0.7690103611460438 _runtime‚4479.762054259 0loss/FRt2/assignment_loss‚0.6991375404419286 1loss/SRqqt2/detection_loss‚0.6566855124550717 /loss/FBt2/detection_loss‚0.6388241401325692†w—À¹œ¨‚(:‘ ¿ºíÒ¿ª gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚179.307' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.861399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Ûã©‚(:Ø ¿ºíÒýµúŽ disk.dm-1.out‚ 133.05859375! disk.nvme0n1.in‚ 5948.38671875* proc.memory.percent‚15.936517715454102# disk.nvme0c0n1.in‚ 5948.38671875 disk.sda2.out‚0 disk.sda3.out‚ 133.05859375 disk.sda.out‚ 133.05859375' proc.memory.availableMB‚ 213.2265625 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.70703125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2611.0390625 cpu‚26.873399024987823# disk./.usageGB‚594.151668548584& disk.nvme0c0n1.out‚10637.126953125 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 1.2906221e+07! disk.dm-0.out‚10637.126953125 disk.sda.in‚0.109375 disk.sda3.in‚0.109375 disk.drbd2165.out‚0$ disk.nvme0n1.out‚10637.126953125 disk.sda1.out‚0 network.recv‚ 1.405021e+06 disk.dm-0.in‚ 5948.38671875 disk.sda1.in‚0$ memory_percent‚98.69856834411621) disk./.usagePercent‚39.87401700275063n"lœª‚(:‘ κíÒÂÈÕŠ gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚136.155' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.230999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á‚@üã«‚(:Ø κíÒÿŸ† disk.sda1.in‚0 disk.dm-1.out‚ 133.640625& disk.nvme0n1.out‚11595.18603515625 disk.rbd0.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚11595.18603515625 network.sent‚ 1.2969016e+07 disk.drbd2165.out‚0 cpu‚26.871467717908995 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚15.936517715454102 disk.sda1.out‚0 disk.sda3.out‚ 133.640625 disk.sda.out‚ 133.640625! disk.nvme0n1.in‚ 5948.38671875$ memory_percent‚98.70138168334961) disk./.usagePercent‚39.87477708911978 disk.rbd0.in‚0 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0 disk.rbd1.out‚ 10.76953125# disk.nvme0c0n1.in‚ 5948.38671875 network.recv‚ 1.409095e+06# disk.dm-0.out‚11595.18603515625 disk.sda3.in‚0.109375& proc.memory.availableMB‚ 212.765625" proc.memory.rssMB‚ 2611.0390625 proc.cpu.threads‚17$ disk./.usageGB‚594.1629943847656 disk.dm-0.in‚ 5948.38671875 disk.sda.in‚0.109375š7”2¬- _wandbruntime‚4501  _runtime‚4501Ùœ Ç­Á & lr-AdamW/pg1‚0.0007471924028547154 & lr-AdamW/pg2‚0.0007471924028547154 trainer/global_step‚9599 % _timestamp‚1.7843725598930066e+09 _runtime‚4501.337010096 _step‚775ŠAÐCÌ®Æ _step‚775 & lr-AdamW/pg1‚0.0007471924028547154 & lr-AdamW/pg2‚0.0007471924028547154 trainer/global_step‚9599 % _timestamp‚1.7843725598930066e+09 _runtime‚4501.337010096‡rí®Þ2¯- _wandbruntime‚4501  _runtime‚4501ë_Ø °ÎO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5841505117184 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162934489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5653891085824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21742334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5633982682022 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372498627315200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366304131388584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91039111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351067200513024 % _timestamp‚1.7843725598942673e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91029134 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366844736229376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366844736229376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚360670062569472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345225695395840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71905355 @8DeviceStatsMonitor.on_train_batch_starÜF:ôüt/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91029134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71905357 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5653979022848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65940562 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚372498732291584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚4501.338250139 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366844753268736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162944468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65940317 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5634068819112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5653891085824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345225720299520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _step‚776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71905355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372498732291584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5653979022848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44198228 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366304028212134ÇAÙ'±ÓO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5634068819112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5653891085824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345225720299520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _step‚776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71905355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372498732291584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5653979022848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44198228 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366304028212134 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5841505117184 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162934489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5653891085824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21742334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5633982682022 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372498627315200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366304131388584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91039111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351067200513024 % _timestamp‚1.7843725598942673e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91029134 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366844736229376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366844736229376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚360670062569472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345225695395840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71905355 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91029134 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71905357 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5653979022848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65940562 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚372498732291584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _runtime‚4501.338250139 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366844753268736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162944468 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65940317 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345260606910464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360706517565440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372536307638784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71912576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71912578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5842269069312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91048406 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360706534604800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162960984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345260631814144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366341067568992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71912576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366881864232960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5842094869504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65946918 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162960984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚9599 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5654460445184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21744209 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5634636140754 # _timestamp‚1.78437256023261e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366881847193600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162950991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21744438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366341170745554 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _runtime‚4501.676765233 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351102900883456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5654460445184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91048406 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44202712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91038415 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345260606910464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360706517565440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372536307638784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71912576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71912578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5842269069312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91048406 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360706534604800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162960984 J;DeviceStatsMonito‡Öçr.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345260631814144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366341067568992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71912576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366881864232960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5842094869504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65946918 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162960984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚9599 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5654460445184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21744209 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5634636140754 # _timestamp‚1.78437256023261e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366881847193600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162950991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21744438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366341170745554 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _runtime‚4501.676765233 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351102900883456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‰v7"2µ- _wandbruntime‚4501  _runtime‚4501è§QÚ ¶ƒ _runtime‚4501.677659413 _step‚778 epoch‚11 0loss/FRt1/assignment_loss‚1.5070541566610336 2loss/SRqqt1/assignment_loss‚0.6008762460316797 /loss/FBt2/detection_loss‚0.5388873061875544 /loss/FRt1/detection_loss‚0.6059848654162124 trainer/global_step‚9599 $ _timestamp‚1.784372560234028e+09 %loss/total_loss‚7.540886407575705 0loss/FRt2/assignment_loss‚0.6693926045904944 0loss/FBt1/assignment_loss‚0.0032000454838111 1loss/SRqqt1/detection_loss‚0.7386196522042155 1loss/SRqqt2/detection_loss‚0.5922358700464296 /loss/FBt1/detection_loss‚0.5199704773437518 3loss/SRqqt2/assignment_loss‚0.47191139369042345 3loss/FBt2/assignment_loss‚0.0003959141440544746 /loss/FRt2/detection_loss‚1.29235787577604577\·ˆ 3loss/FBt2/assignment_loss‚0.0003959141440544746 /loss/FRt2/detection_loss‚1.2923578757760457 _runtime‚4501.677659413 _step‚778 epoch‚11 0loss/FRt1/assignment_loss‚1.5070541566610336 2loss/SRqqt1/assignment_loss‚0.6008762460316797 /loss/FBt2/detection_loss‚0.5388873061875544 /loss/FRt1/detection_loss‚0.6059848654162124 trainer/global_step‚9599 $ _timestamp‚1.784372560234028e+09 %loss/total_loss‚7.540886407575705 0loss/FRt2/assignment_loss‚0.6693926045904944 0loss/FBt1/assignment_loss‚0.0032000454838111 1loss/SRqqt1/detection_loss‚0.7386196522042155 1loss/SRqqt2/detection_loss‚0.5922358700464296 /loss/FBt1/detection_loss‚0.5199704773437518 3loss/SRqqt2/assignment_loss‚0.47191139369042345ŠQT}Wœ¸‚(:‘ ݺíÒ ª¾‹ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚190.712' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.142399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èv. 繂(:Ü ݺíÒúçä’ disk.sda3.out‚ 134.8046875 disk.rbd1.out‚ 10.76953125 disk.sda1.in‚0) proc.memory.percent‚16.24748706817627# disk.dm-0.out‚11749.60888671875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚11749.60888671875! disk.nvme0n1.in‚ 6122.18359375 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 162.19140625 disk.sda2.out‚0 disk.dm-0.in‚ 6122.18359375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 6122.18359375 disk.sda1.out‚0# memory_percent‚99.0100622177124$ disk./.usageGB‚594.0710830688477 disk.sda2.in‚0 disk.dm-1.out‚ 134.8046875& disk.nvme0n1.out‚11749.60888671875# proc.memory.rssMB‚ 2661.98828125 network.sent‚ 1.298703e+07 network.recv‚ 1.412305e+06 disk.drbd2165.in‚0 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375 cpu‚26.876554014898446* disk./.usagePercent‚39.868608843589804 disk.sda3.in‚0.109375 disk.sda.out‚ 134.8046875 proc.cpu.threads‚17\øh02º- _wandbruntime‚4526  _runtime‚4526î”â¹=»³{ & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.845838725566864 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1398656666278839 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16009776294231415 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025344636291265488 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5144604444503784 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037554774433374405 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8181013464927673 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25578412413597107 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14551933109760284 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0082757119089365 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1860523223876953 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5815588235855103 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.651538610458374 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5393499135971069 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6197097301483154 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9425646662712097 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6241785287857056 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5815097093582153 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9757275581359863 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11117447912693024 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14551933109760284 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.171392560005188 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.544646143913269 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5804859399795532 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5685261487960815 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6704171299934387 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.144312784075737 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46590909361839294 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_4‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7679486870765686 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4992613196372986 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14623628556728363 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43973368406295776 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16167522966861725 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_2_of_2‚0.13190139830112457 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5384923815727234 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5642101764678955 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7280271649360657 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831498265266418 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6677085161209106 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6544514298439026 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1682061105966568 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21800705790519714 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878638386726379 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15343505144119263 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991559624671936 6validation_average_jet_accuracy‚0.3169352114200592 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8359649181365967 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897671937942505 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015425930730998516 particle/accuracy_1_of_1‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8585603833198547 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5376137495040894 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.32105809450149536 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949615001678467 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7215896844863892 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6489077806472778 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5101548433303833 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6012927293777466 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1940499246120453 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5834414958953857 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9331150054931641 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.572486162185669 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17887593805789948 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 'particle/f_score‚0.6332834362983704 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985022246837616 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.917387843132019 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1239437460899353 ,jet/accuracy_4_of_4‚0.014682539738714695 +particle/specificity‚0.9076168537139893 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998918890953064 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15123559534549713 epoch‚11 )loss/val_total_loss‚7.975283863451409 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956019520759583 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.730667233467102 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6343117952346802 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3103432059288025 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17517870664596558 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7215896844863892 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1653156727552414 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.606643557548523 particle/accuracy_1_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5421633124351501 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3122556209564209 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.488627165555954 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.663786768913269 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966767430305481 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8784244060516357 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1510932445526123 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7490500211715698 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6666980385780334 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5447916984558105 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3220183849334717 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2136063277721405 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934656023979187 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.623556911945343 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 _step‚779 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7449148297309875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942086338996887 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5393499135971069 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.409722238779068 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995815098285675 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6206926703453064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9000401496887207 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5169617533683777 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 +jet/accuracy_3_of_4‚0.09337560832500458 +particle/sensitivity‚0.6199457049369812 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12757335603237152 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.31644225120544434 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12733270227909088 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6274707317352295 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.634306788444519 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47999998927116394 particle/accuracy_2_of_4‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12729722261428833 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5835649371147156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12797358632087708 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12763717770576477 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6119347214698792 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12729722261428833 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.034157298505306244 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22376810014247894 jet/accuracy_4_of_5‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42901232838630676 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.26795464754104614 *jet/accuracy_1_of_2‚0.7230624556541443 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23865579068660736 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.621871829032898 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8222221732139587 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5664056539535522 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_3‚0.9008221626281738 trainer/global_step‚9611 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_3_of_3‚0.07568373531103134 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8359649181365967 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01843687705695629 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6631944179534912 % _timestamp‚1.7843725846386628e+09 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5823955535888672 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17246614396572113 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14172297716140747 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5462830662727356 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 _runtime‚4526.083205242 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19135497510433197 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9583487510681152 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.843796968460083 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15953418612480164°çKý##¼¸{ :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14172297716140747 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5462830662727356 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 _runtime‚4526.083205242 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19135497510433197 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9583487510681152 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.843796968460083 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15953418612480164 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.845838725566864 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1398656666278839 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16009776294231415 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025344636291265488 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5144604444503784 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037554774433374405 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8181013464927673 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25578412413597107 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14551933109760284 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0082757119089365 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1860523223876953 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5815588235855103 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.651538610458374 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5393499135971069 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6197097301483154 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9425646662712097 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6241785287857056 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5815097093582153 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9757275581359863 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11117447912693024 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14551933109760284 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.171392560005188 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.544646143913269 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5804859399795532 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5685261487960815 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6704171299934387 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.144312784075737 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46590909361839294 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_4‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7679486870765686 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4992613196372986 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14623628556728363 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43973368406295776 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16167522966861725 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_2_of_2‚0.13190139830112457 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5384923815727234 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5642101764678955 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7280271649360657 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831498265266418 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6677085161209106 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6544514298439026 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1682061105966568 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21800705790519714 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878638386726379 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15343505144119263 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991559624671936 6validation_average_jet_accuracy‚0.3169352114200592 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8359649181365967 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897671937942505 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015425930730998516 particle/accuracy_1_of_1‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8585603833198547 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5376137495040894 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.32105809450149536 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949615001678467 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7215896844863892 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6489077806472778 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5101548433303833 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6012927293777466 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1940499246120453 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5834414958953857 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9331150054931641 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.572486162185669 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17887593805789948 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 'particle/f_score‚0.6332834362983704 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985022246837616 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.917387843132019 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1239437460899353 ,jet/accuracy_4_of_4‚0.014682539738714695 +particle/specificity‚0.9076168537139893 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998918890953064 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15123559534549713 epoch‚11 )loss/val_total_loss‚7.975283863451409 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956019520759583 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.730667233467102 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.63431179523468|ê›02 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3103432059288025 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17517870664596558 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7215896844863892 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1653156727552414 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.606643557548523 particle/accuracy_1_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5421633124351501 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3122556209564209 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.488627165555954 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.663786768913269 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966767430305481 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8784244060516357 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1510932445526123 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7490500211715698 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6666980385780334 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5447916984558105 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3220183849334717 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2136063277721405 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9934656023979187 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.623556911945343 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 _step‚779 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7449148297309875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942086338996887 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5393499135971069 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.409722238779068 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995815098285675 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6206926703453064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9000401496887207 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5169617533683777 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 +jet/accuracy_3_of_4‚0.09337560832500458 +particle/sensitivity‚0.6199457049369812 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12757335603237152 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.31644225120544434 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12733270227909088 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6274707317352295 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.634306788444519 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47999998927116394 particle/accuracy_2_of_4‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12729722261428833 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5835649371147156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12797358632087708 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12763717770576477 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6119347214698792 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12729722261428833 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.034157298505306244 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22376810014247894 jet/accuracy_4_of_5‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42901232838630676 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.26795464754104614 *jet/accuracy_1_of_2‚0.7230624556541443 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23865579068660736 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.621871829032898 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8222221732139587 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5664056539535522 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_3‚0.9008221626281738 trainer/global_step‚9611 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +jet/accuracy_3_of_3‚0.07568373531103134 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8359649181365967 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01843687705695629 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6631944179534912 % _timestamp‚1.7843725846386628e+09 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5823955535888672 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17246614396572113‹É,éX›½‚(: ìºíÒ®€  gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚171.312' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.26240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06澂(:Û ìºíÒòÚ° disk.sda.out‚ 136.5859375! disk.nvme0n1.in‚ 6122.18359375' proc.memory.availableMB‚ 148.2734375 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.drbd2165.out‚0 network.sent‚ 1.3049897e+07# disk./.usageGB‚593.624813079834 disk.rbd0.out‚0 disk.sda3.in‚0.109375 disk.sda3.out‚ 136.5859375& disk.nvme0n1.out‚11863.12646484375 disk.rbd0.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6122.18359375* proc.memory.percent‚16.331791877746582 disk.dm-1.out‚ 136.5859375 disk.rbd1.out‚ 10.83203125( disk.nvme0c0n1.out‚11863.12646484375$ memory_percent‚99.09501075744629) disk./.usagePercent‚39.83865929018163 disk.dm-0.in‚ 6122.18359375# disk.dm-0.out‚11863.12646484375# proc.memory.rssMB‚ 2675.80078125 cpu‚26.884297851760177 network.recv‚ 1.41667e+06 disk.sda2.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.109375Àì$2¿- _wandbruntime‚4542  _runtime‚4542NP6¢ÆÀÀ $ _timestamp‚1.784372600998611e+09 _runtime‚4542.442505158 _step‚780 & lr-AdamW/pg1‚0.0007460911031583864 & lr-AdamW/pg2‚0.0007460911031583864 trainer/global_step‚9649ïŒzÃËÁÅ & lr-AdamW/pg2‚0.0007460911031583864 trainer/global_step‚9649 $ _timestamp‚1.784372600998611e+09 _runtime‚4542.442505158 _step‚780 & lr-AdamW/pg1‚0.0007460911031583864Œ³µ2Â- _wandbruntime‚4542  _runtime‚4542ÊáÔ'ÃÎO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5887506001408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375360168960000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5678869263171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164149521 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 trainer/global_step‚9649 _step‚781 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72500933 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91648586 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4542.443776023 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72500935 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91638609 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353773290552832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347885784551424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚375360168960000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 % _timestamp‚1.7843726009997625e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21924470 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369661280833536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72500935 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44527005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5698993102848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5698905165824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5698993102848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5698905165824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5678955400261 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369129837610053 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347885809455104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369661263794176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66451713 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66451475 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164139542 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369661263794176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5887676014080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44527008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚363450882209792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369661280833536 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91648586 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚375360273936384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91638609 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72500933 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353773485469184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚375360168960000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 % _timestamp‚1.7843726009997625e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21924470 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369661280833536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72500935 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44527005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5698993102848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5698905165824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5698993102848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5698905165824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5678955400261 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369129837610053 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347885809455104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369661263794176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66451713 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66451475 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164139542 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369661263794176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5887676014080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44527008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚363450882209792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369661280833536 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91648586 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚375360273936384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91638609 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72500933 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353773485469184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375360168960000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5678869263171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164149521 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 trainer/global_step‚9649 _step‚781 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72500933 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91648586 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4542.443776023 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72500935 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91638609 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353773290552832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347885784551424HÀß2Å- _wandbruntime‚4542  _runtime‚4542"AÑà ÆØM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72508156 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66458068 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5888095966720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚363487354245120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚9649 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚363487337205760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369698374758400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 569SæD9562469376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66458313 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347920696066048 _step‚782 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369698391797760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72508156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21926579 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353808991136256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843726013363562e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375397954267136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164166037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4542.780401499 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369698391797760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5888270166528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44531489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91657881 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91657881 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369166876967023 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44531492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21926821 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164156044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5679436584701 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369698391797760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72508156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21926579 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353808991136256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843726013363562e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375397954267136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164166037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4542.780401499 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369698391797760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5888270166528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44531489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91657881 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91657881 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369166876967023 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44531492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21926821 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164156044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5679436584701 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72508156 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66458068 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5888095966720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚363487354245120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚9649 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚363487337205760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369698374758400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562ŽÊ4“a2È- _wandbruntime‚4542  _runtime‚4542dKù$ Ɇ 2loss/SRqqt1/assignment_loss‚0.5127987343431493 3loss/SRqqt2/assignment_loss‚0.41904166960220063 3loss/FBt2/assignment_loss‚0.0005074911820808814 /loss/FRt1/detection_loss‚0.5079015080608306 _runtime‚4542.781983521 .loss/FRt2/detection_loss‚1.504843850902731 trainer/global_step‚9649 _step‚783 0loss/FBt2/detection_loss‚0.48763897574491916 epoch‚12 0loss/FRt1/assignment_loss‚1.6091102564751991 0loss/FRt2/assignment_loss‚0.7821283770437858 2loss/FBt1/assignment_loss‚8.98274379999009e-05 1loss/SRqqt2/detection_loss‚0.6375533597059198 1loss/SRqqt1/detection_loss‚0.7324468730088405 /loss/FBt1/detection_loss‚0.4249662424230741 &loss/total_loss‚7.6190271659307305 $ _timestamp‚1.784372601338425e+093Ê‹ $ _timestamp‚1.784372601338425e+09 2loss/SRqqt1/assignment_loss‚0.5127987343431493 3loss/SRqqt2/assignment_loss‚0.41904166960220063 3loss/FBt2/assignment_loss‚0.0005074911820808814 /loss/FRt1/detection_loss‚0.5079015080608306 _runtime‚4542.781983521 .loss/FRt2/detection_loss‚1.504843850902731 trainer/global_step‚9649 _step‚783 0loss/FBt2/detection_loss‚0.48763897574491916 epoch‚12 0loss/FRt1/assignment_loss‚1.6091102564751991 0loss/FRt2/assignment_loss‚0.7821283770437858 2loss/FBt1/assignment_loss‚8.98274379999009e-05 1loss/SRqqt2/detection_loss‚0.6375533597059198 1loss/SRqqt1/detection_loss‚0.7324468730088405 /loss/FBt1/detection_loss‚0.4249662424230741 &loss/total_loss‚7.6190271659307305)&ºà‘Ë‚(:† ûºíÒ¤ûŒ gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚179.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ˆÑ±ìÌ‚(:á ûºíÒÆ‹ñ‘$ disk./.usageGB‚593.5692710876465 disk.rbd1.in‚0& disk.nvme0n1.out‚12116.44580078125( proc.memory.availableMB‚ 147.86328125 disk.sda.out‚ 136.95703125 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 6122.18359375# disk.dm-0.out‚12116.44580078125 network.sent‚ 1.3094492e+07 disk.sda3.in‚0.109375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12116.44580078125 disk.sda3.out‚ 136.95703125 disk.rbd0.out‚0 disk.dm-1.in‚0.109375 cpu‚26.882295566209827 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6122.18359375 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6122.18359375 network.recv‚ 1.420895e+06 disk.sda.in‚0.109375 disk.sda2.out‚0 disk.dm-1.out‚ 136.95703125# proc.memory.rssMB‚ 2675.80078125* proc.memory.percent‚16.331791877746582) disk./.usagePercent‚39.83493182048309 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.83203125$ memory_percent‚99.09751415252686üþýsÍ‚(:„ Š»íÒƒ’ØŠ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚186.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eºâ΂(:× Š»íÒÃÔºŒ disk.dm-1.out‚ 138.46875 disk.dm-0.in‚ 6122.18359375 disk.sda3.out‚ 138.46875 disk.drbd2165.in‚0 network.sent‚ 1.3157372e+07 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.82979323725582$ disk./.usageGB‚593.4927024841309 disk.sda1.in‚0 cpu‚26.880240144962265 network.recv‚ 1.425321e+06( disk.nvme0c0n1.out‚12289.97900390625* proc.memory.percent‚16.331791877746582$ memory_percent‚99.09682273864746 disk.sda.in‚0.109375 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# disk.dm-0.out‚12289.97900390625 disk.sda.out‚ 138.46875 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.89453125' proc.memory.availableMB‚ 147.9765625# proc.memory.rssMB‚ 2675.80078125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6122.18359375! disk.nvme0n1.in‚ 6122.18359375& disk.nvme0n1.out‚12289.97900390625.®KÀ2Ï- _wandbruntime‚4564  _runtime‚4564Œø{´ÇÐÁ _step‚784 & lr-AdamW/pg1‚0.0007449846450055383 & lr-AdamW/pg2‚0.0007449846450055383 trainer/global_step‚9699 % _timestamp‚1.7843726226911688e+09 _runtime‚4564.134980176:¨EÌÑÆ & lr-AdamW/pg2‚0.0007449846450055383 trainer/global_step‚9699 % _timestamp‚1.7843726226911688e+09 _runtime‚4564.134980176 _step‚784 & lr-AdamW/pg1‚0.0007449846450055383ц¸&2Ò- _wandbruntime‚4564  _runtime‚4564î&Ù#Ô'ÓÎO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44751208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92114488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365274466936832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22033700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚4564.136239333 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377244322372608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92104511 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377244322372608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355548463414784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92104511 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66784908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22033443 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5916899827200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5727405390848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355548260109824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349631385186304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72861983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371516829044736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164976473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72861985 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5917078228480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚785 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370981734179711 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5727493327872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5727493327872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72861985 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92114488 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377244217396224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370981837356161 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0åœg‘ÔÓO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72861985 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5917078228480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚785 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370981734179711 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5727493327872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5727493327872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72861985 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92114488 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377244217396224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370981837356161 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44751208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92114488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365274466936832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22033700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚4564.136239333 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377244322372608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92104511 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377244322372608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355548463414784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92104511 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66784908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22033443 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5916899827200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5727405390848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355548260109824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349631385186304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72861983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371516829044736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164976473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488‘*GÒ<2Õ- _wandbruntime‚4564  _runtime‚4564n#—Þ&ÖØM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164992989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66791335 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164982996 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371553940008960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92113792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚371018773536569 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365310938972160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 trainer/global_step‚9699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164992989 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5707920701611 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72869204 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66791567 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚371018876713131 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚371553922969600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377282002703360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72869204 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164982996 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44755692 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚355583963372032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5707834564409 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 % _timestamp‚1.7843726230328794e+09 _runtime‚4564.476972195 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365310956011520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355583762171392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349666296700928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5917490374144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚786!)hã&×ÝM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92123783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22035646 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92123783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44755689 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72869204 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164982996 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44755692 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚355583963372032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5707834564409 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 % _timestamp‚1.7843726230328794e+09 _runtime‚4564.476972195 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365310956011520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355583762171392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349666296700928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5917490374144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚786 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164992989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66791335 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164982996 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371553940008960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92113792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚371018773536569 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365310938972160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 trainer/global_step‚9699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164992989 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5707920701611 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72869204 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66791567 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚371018876713131 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚371553922969600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377282002703360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379128265832448 M²œ-'^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755993552896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351376960917504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44975408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946500024832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22142178 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚373372360216576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73223033 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92570413 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372833733925819 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92570413 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73223035 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165793446 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73223033 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843726444949334e+09 _step‚789 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5735751359677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367098085742592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67117586 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5755993552896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373372377255936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22141940 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379128370808832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379128265832448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357323460942336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92580390 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327¬K‚)Ù'âÓO % _timestamp‚1.7843726444949334e+09 _step‚789 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5735751359677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367098085742592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67117586 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5755993552896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373372377255936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22141940 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379128370808832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379128265832448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357323460942336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92580390 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379128265832448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755993552896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351376960917504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44975408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946500024832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22142178 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚373372360216576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73223033 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92570413 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372833733925819 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92570413 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73223035 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165793446 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73223033 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880•šýݲ2ã- _wandbruntime‚4586  _runtime‚4586J:»Þ&äØM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22144298 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5756474975232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351411847528448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357358962943488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843726448338504e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165819941 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73230256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367134540738560 _step‚790 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165809948 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373409471180800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44979889 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73230254 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379166051139584 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367134557777920 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372870773282677 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92579694 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5756562919424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5756474975232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73230256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5947090511360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379165946156032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379166051139584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73230254 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5736318681319 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92589685 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373409488220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165819941 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67123946 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351411872432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5756562919424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92579694 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165819941 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67123946 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351411872432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5756562919424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92579694 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5756474975232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351411847528448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357358962943488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843726448338504e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165819941 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73230256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367134540738560 _step‚790 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165809948 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373409471180800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44979889 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73230254 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379166051139584 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367134557777920 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372870773282677 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92579694 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5756562919424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5756474975232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73230256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5947090511360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379165946156032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379166051139584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73230254 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5736318681319 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92589685 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202–dvm2æ- _wandbruntime‚4586  _runtime‚4586£Óhß ç„ 3loss/FBt2/assignment_loss‚0.0020187764663060475 %loss/total_loss‚7.705355508830304 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚0.6775536981393706 3loss/SRqqt1/assignment_loss‚0.49016582575499595 /loss/FRt1/detection_loss‚0.5439161757714319 /loss/FRt2/detection_loss‚1.4437690721912626 1loss/SRqqt2/detection_loss‚0.7345573320154749 /loss/FBt2/detection_loss‚0.4610115914873666 epoch‚12 $ _timestamp‚1.784372644835488e+09 2loss/FBt1/assignment_loss‚0.059429018294218136 /loss/FBt1/detection_loss‚0.3363695382606238 _runtime‚4586.279029076 _step‚791 1loss/SRqqt1/detection_loss‚0.8406687560914592 /loss/FRt1/assignment_loss‚1.611469500899315 2loss/SRqqt2/assignment_loss‚0.5044262234584793ŠFšè‰ 2loss/SRqqt2/assignment_loss‚0.5044262234584793 3loss/FBt2/assignment_loss‚0.0020187764663060475 %loss/total_loss‚7.705355508830304 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚0.6775536981393706 3loss/SRqqt1/assignment_loss‚0.49016582575499595 /loss/FRt1/detection_loss‚0.5439161757714319 /loss/FRt2/detection_loss‚1.4437690721912626 1loss/SRqqt2/detection_loss‚0.7345573320154749 /loss/FBt2/detection_loss‚0.4610115914873666 epoch‚12 $ _timestamp‚1.784372644835488e+09 2loss/FBt1/assignment_loss‚0.059429018294218136 /loss/FBt1/detection_loss‚0.3363695382606238 _runtime‚4586.279029076 _step‚791 1loss/SRqqt1/detection_loss‚0.8406687560914592 /loss/FRt1/assignment_loss‚1.611469500899315—Yëý‘é‚(:† ¨»íÒ·šŠ gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚138.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}vxèê‚(:Ý ¨»íÒ˜éÓŒ network.sent‚ 1.3225027e+07 disk.dm-1.out‚ 142.30078125 disk.dm-0.in‚ 6205.92578125 disk.drbd2165.out‚0% disk.nvme0n1.out‚12375.7666015625 disk.sda3.in‚0.109375 disk.sda.in‚0.109375# proc.memory.rssMB‚ 2675.80078125* proc.memory.percent‚16.331791877746582 network.recv‚ 1.432154e+06# disk.nvme0c0n1.in‚ 6205.92578125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0" disk.dm-0.out‚12375.7666015625$ memory_percent‚99.09653663635254 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 10.95703125 disk.sda.out‚ 142.30078125) disk./.usagePercent‚39.82949422045583$ disk./.usageGB‚593.4882469177246 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 142.30078125' disk.nvme0c0n1.out‚12375.7666015625! disk.nvme0n1.in‚ 6205.92578125' proc.memory.availableMB‚ 148.0234375 disk.sda2.in‚0 disk.dm-1.in‚0.109375 cpu‚26.8761432309211126{«É›ë‚(: ·»íÒ½¬Š gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.724000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b±sêì‚(:ß ·»íÒÁÛñŒ# proc.memory.rssMB‚ 2675.80078125* disk./.usagePercent‚39.829493196425695! disk.nvme0n1.in‚ 6205.92578125 disk.rbd0.in‚0# disk.dm-0.out‚12380.63037109375 disk.dm-0.in‚ 6205.92578125 disk.sda3.out‚ 142.6640625 cpu‚26.874113024684785& disk.nvme0n1.out‚12380.63037109375 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 network.sent‚ 1.3289845e+07 disk.rbd1.out‚ 10.95703125# disk.nvme0c0n1.in‚ 6205.92578125* proc.memory.percent‚16.331791877746582 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 148.08203125$ disk./.usageGB‚593.4882316589355 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚12380.63037109375$ memory_percent‚99.09617900848389 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0 network.recv‚ 1.437711e+06 disk.sda2.out‚0 disk.dm-1.out‚ 142.6640625 disk.sda.out‚ 142.6640625Ûå}B2í- _wandbruntime‚4607  _runtime‚46076ȧñÇîÁ & lr-AdamW/pg1‚0.0007427563243013173 & lr-AdamW/pg2‚0.0007427563243013173 trainer/global_step‚9799 % _timestamp‚1.7843726661864784e+09 _runtime‚4607.630274452 _step‚792eNM~ÌïÆ & lr-AdamW/pg1‚0.0007427563243013173 & lr-AdamW/pg2‚0.0007427563243013173 trainer/global_step‚9799 % _timestamp‚1.7843726661864784e+09 _runtime‚4607.630274452 _step‚792˜$ÍŽ¤2ð- _wandbruntime‚4607  _runtime‚4607µ§2?Ó'ñÍO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375227925467136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381012419245056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22251741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359098311705088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73584085 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73584083 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5784405840896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45199605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368921670469632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67451576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374685836848377 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5764063202295 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5784405840896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353122511745024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375227908427776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5975799960064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375227925467136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 $ _timestamp‚1.784372666187547e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67451346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93036315 _runtime‚4607.631524691 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166630377 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93046292 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5764149339385 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45199608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381012314268672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22251968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381012314268672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22251968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381012419245056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22251741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359098311705088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73584085 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73584083 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5784405840896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45199605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368921670469632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67451576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374685836848377 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5764063202295 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5784405840896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚230 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚227 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353122511745024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375227908427776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5975799960064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375227925467136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 $ _timestamp‚1.784372666187547e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67451346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93036315 _runtime‚4607.631524691 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166630377 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93046292 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5764149339385 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ?9DeviceStatsMonitor.on_trainˆRî–i_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473™Pºî2ó- _wandbruntime‚4607  _runtime‚4607$VhÞ&ôØM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚381050099575808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5785063144448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5784975200256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67457941 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45204089 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843726665266569e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368958159544320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381050099575808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166646893 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93045596 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 _runtime‚4607.970716189 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359133809560064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚9799 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5976562597376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374722876205347 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93055587 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22253852 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5976386300416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381049994592256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359134010760704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5764630523825 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375265019392000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166636900 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93045596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93055587 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73591304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353157423259648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374722773028785 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166636900 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375265019392000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73591306 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5785063144448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166646893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936ô#~:ã&õÝM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93045596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93055587 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73591304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353157423259648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374722773028785 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166636900 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375265019392000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73591306 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5785063144448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166646893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚381050099575808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5785063144448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5784975200256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67457941 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45204089 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 % _timestamp‚1.7843726665266569e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368958159544320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381050099575808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166646893 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93045596 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 _runtime‚4607.970716189 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359133809560064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚9799 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5976562597376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374722876205347 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93055587 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22253852 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5976386300416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381049994592256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359134010760704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5764630523825 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375265019392000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166636900 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146šÌ#¼12ö- _wandbruntime‚4607  _runtime‚4607‘Dt÷ˆ /loss/FRt1/assignment_loss‚1.692210720793051 2loss/SRqqt1/assignment_loss‚0.5540922649335741 epoch‚12 _runtime‚4607.971893742 _step‚795 /loss/FRt2/detection_loss‚1.5395087930501077 1loss/SRqqt1/detection_loss‚0.7185140472853131 1loss/SRqqt2/detection_loss‚0.6099008581233382 2loss/SRqqt2/assignment_loss‚0.4884805117342627 /loss/FBt1/detection_loss‚0.5562612545366089 %loss/total_loss‚7.770244640629228 trainer/global_step‚9799 % _timestamp‚1.7843726665283554e+09 0loss/FRt2/assignment_loss‚0.5834929252582821 4loss/FBt1/assignment_loss‚0.00010279306924256085 4loss/FBt2/assignment_loss‚0.00025140953307593996 /loss/FRt1/detection_loss‚0.5803061523274902 0loss/FBt2/detection_loss‚0.44712290998488186ÖÉ×ø 4loss/FBt2/assignment_loss‚0.00025140953307593996 /loss/FRt1/detection_loss‚0.5803061523274902 0loss/FBt2/detection_loss‚0.44712290998488186 /loss/FRt1/assignment_loss‚1.692210720793051 2loss/SRqqt1/assignment_loss‚0.5540922649335741 epoch‚12 _runtime‚4607.971893742 _step‚795 /loss/FRt2/detection_loss‚1.5395087930501077 1loss/SRqqt1/detection_loss‚0.7185140472853131 1loss/SRqqt2/detection_loss‚0.6099008581233382 2loss/SRqqt2/assignment_loss‚0.4884805117342627 /loss/FBt1/detection_loss‚0.5562612545366089 %loss/total_loss‚7.770244640629228 trainer/global_step‚9799 % _timestamp‚1.7843726665283554e+09 0loss/FRt2/assignment_loss‚0.5834929252582821 4loss/FBt1/assignment_loss‚0.00010279306924256085›€ùœù‚(:‘ Æ»íÒÄžŒ gpu.0.gpu‚65.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚126.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.293599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_«ïâú‚(:× Æ»íÒ¼¤¾ cpu‚26.872109359664737 disk.rbd0.in‚0 disk.dm-1.in‚0.109375 disk.sda3.out‚ 143.015625% disk.nvme0n1.out‚12388.4775390625 disk.rbd1.in‚0 disk.rbd1.out‚ 11.01953125# proc.memory.rssMB‚ 2675.80078125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚12388.4775390625 network.recv‚ 1.441089e+06 disk.dm-1.out‚ 143.015625 disk.sda.out‚ 143.015625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 6205.92578125" disk.dm-0.out‚12388.4775390625# disk.nvme0c0n1.in‚ 6205.92578125' proc.memory.availableMB‚ 148.2734375 network.sent‚ 1.3293612e+07! disk.nvme0n1.in‚ 6205.92578125* proc.memory.percent‚16.331791877746582) disk./.usagePercent‚39.83069719985922 disk.sda.in‚0.109375 disk.drbd2165.out‚0$ memory_percent‚99.09501075744629$ disk./.usageGB‚593.5061721801758 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375ÑÒBg2û- _wandbruntime‚4629  _runtime‚4629K«hAÆüÀ & lr-AdamW/pg1‚0.0007416344974410235 & lr-AdamW/pg2‚0.0007416344974410235 trainer/global_step‚9849 % _timestamp‚1.7843726880858812e+09 _runtime‚ 4629.52968459 _step‚796D P£ËýÅ trainer/global_step‚9849 % _timestamp‚1.7843726880858812e+09 _runtime‚ 4629.52968459 _step‚796 & lr-AdamW/pg1‚0.0007416344974410235 & lr-AdamW/pg2‚0.0007416344974410235œMKé¸2þ- _wandbruntime‚4629  _runtime‚4629Ñ·¦x|#ÿÍO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5812906065920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370745289275392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67784169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382896467681280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382896362704896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5812994002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167457329 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376537733418035 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382896362704896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382896467681280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45423805 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 $ _timestamp‚1.784372688086958e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73945133 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377083473678336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93502217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22360602 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377083456638976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5792461182003 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22360364 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360873335110656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377083473678336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354868112379904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6005247634432 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large¶|ˆOW_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45423808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5792547319093 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 J·ÅèØ'€ÒO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22360364 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360873335110656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377083473678336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354868112379904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6005247634432 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45423808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5792547319093 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67784169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382896467681280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382896362704896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5812994002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167457329 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376537733418035 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382896362704896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382896467681280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45423805 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 $ _timestamp‚1.784372688086958e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73945133 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377083473678336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93502217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22360602 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377083456638976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5792461182003 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144Mx¬Î2- _wandbruntime‚4629  _runtime‚4629–_ öÞ&‚ØM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813563369472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73952354 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382934043028480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚798 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73952354 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67790830 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5813563369472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167463852 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5793114640735 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6005830513664 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167473845 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93521489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167473845 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93511498 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377120584642560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6006015199232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5813475425280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73952356 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚9849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67791070 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22362541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370781761310720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45428292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45428289 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67790830 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5813563369472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167463852 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5793114640735 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6005830513664 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167473845 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93521489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167473845 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93511498 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377120584642560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6006015199232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5813475425280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73952356 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 trainer/global_step‚9849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67791070 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22362541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370781761310720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45428292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45428289 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73952354 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360žÑá2„- _wandbruntime‚4629  _runtime‚4629æG4U …‡ /loss/FRt1/detection_loss‚0.6013711975035423 .loss/FRt2/detection_loss‚1.509086201033075 1loss/SRqqt2/detection_loss‚0.6019520387066948 %loss/total_loss‚7.969342673088057 0loss/FRt1/assignment_loss‚1.6108319424684086 2loss/SRqqt2/assignment_loss‚0.4271186609178853 /loss/FBt1/detection_loss‚0.6950886253775521 % _timestamp‚1.7843726884299293e+09 0loss/FRt2/assignment_loss‚0.6382752100276526 0loss/FBt2/detection_loss‚0.38706391896416864 epoch‚12 2loss/SRqqt1/assignment_loss‚0.6209933924262545 4loss/FBt1/assignment_loss‚0.00014903282459919544 1loss/SRqqt1/detection_loss‚0.8759695564849036 trainer/global_step‚9849 _runtime‚4629.873483981 _step‚799 3loss/FBt2/assignment_loss‚0.0014428963533215699d¨)Šg†Œ 2loss/SRqqt1/assignment_loss‚0.6209933924262545 4loss/FBt1/assignment_loss‚0.00014903282459919544 1loss/SRqqt1/detection_loss‚0.8759695564849036 trainer/global_step‚9849 _runtime‚4629.873483981 _step‚799 3loss/FBt2/assignment_loss‚0.0014428963533215699 /loss/FRt1/detection_loss‚0.6013711975035423 .loss/FRt2/detection_loss‚1.509086201033075 1loss/SRqqt2/detection_loss‚0.6019520387066948 %loss/total_loss‚7.969342673088057 0loss/FRt1/assignment_loss‚1.6108319424684086 2loss/SRqqt2/assignment_loss‚0.4271186609178853 /loss/FBt1/detection_loss‚0.695íXíð«0886253775521 % _timestamp‚1.7843726884299293e+09 0loss/FRt2/assignment_loss‚0.6382752100276526 0loss/FBt2/detection_loss‚0.38706391896416864 epoch‚12Ÿß] º‘‡‚(:† Õ»íÒîÝ­Š gpu.0.gpu‚62.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚181.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ó(Q눂(:à Õ»íÒÀÿ÷Œ disk.sda.in‚0.109375 disk.drbd2165.out‚0# disk.dm-0.out‚12482.41162109375* disk./.usagePercent‚39.830684143474976$ disk./.usageGB‚593.5059776306152& disk.nvme0n1.out‚12482.41162109375# proc.memory.rssMB‚ 2675.80078125* proc.memory.percent‚16.331791877746582 disk.sda.out‚ 143.29296875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 143.29296875 disk.dm-0.in‚ 6310.16796875$ memory_percent‚99.09398555755615( proc.memory.availableMB‚ 148.44140625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.01953125 disk.sda1.in‚0 cpu‚26.87009684104067 network.sent‚ 1.335592e+07 disk.sda3.in‚0.109375 disk.sda3.out‚ 143.29296875! disk.nvme0n1.in‚ 6310.16796875 network.recv‚ 1.445331e+06# disk.nvme0c0n1.in‚ 6310.16796875( disk.nvme0c0n1.out‚12482.41162109375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17g±‘‰‚(:† ä»íÒŒ©½Š gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚166.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J[⊂(:× ä»íÒÞ‰© disk.sda3.in‚0.109375# disk.nvme0c0n1.in‚ 6310.16796875$ disk.nvme0n1.out‚12486.841796875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 6310.16796875 network.recv‚ 1.449573e+06 disk.sda.in‚0.109375( proc.memory.availableMB‚ 148.02734375 disk.rbd1.out‚ 11.08203125 disk.sda2.in‚0 disk.sda.out‚ 143.6328125! disk.nvme0n1.in‚ 6310.16796875 disk.dm-1.out‚ 143.6328125# proc.memory.rssMB‚ 2675.80078125 proc.cpu.threads‚17) disk./.usagePercent‚39.83059351680785 disk.sda3.out‚ 143.6328125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.09651279449463 disk.drbd2165.out‚0* proc.memory.percent‚16.331791877746582& disk.nvme0c0n1.out‚12486.841796875 network.sent‚ 1.3419127e+07$ disk./.usageGB‚593.5046272277832 disk.drbd2165.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚12486.841796875 cpu‚26.86810381474774 disk.rbd1.in‚0+,2‹- _wandbruntime‚4651  _runtime‚4651Q§:¡ÇŒÁ & lr-AdamW/pg1‚0.0007405075835057972 & lr-AdamW/pg2‚0.0007405075835057972 trainer/global_step‚9899 % _timestamp‚1.7843727100118883e+09 _runtime‚4651.456801624 _step‚800ç&ëÖÌÆ _runtime‚4651.456801624 _step‚800 & lr-AdamW/pg1‚0.0007405075835057972 & lr-AdamW/pg2‚0.0007405075835057972 trainer/global_step‚9899 % _timestamp‚1.7843727100118883e+09 ·š‡À2Ž- _wandbruntime‚4651  _runtime‚4651²üÓÝÔ'ÎO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5820945298801 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378939021889536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5841494227968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74306185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356613663207424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362648558008320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5820859161711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68117131 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384780411141120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚378939021889536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362648358897664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372568891041792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93978096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93968119 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45648008 trainer/global_step‚9899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384780411141120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22469126 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6034869897216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843727100126963e+09 _step‚801 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74306183 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384780516117504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5841494227968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6034695690240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372568874002432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168274302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5841406290944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93978096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74306185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74306183 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45648005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚378389733164143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚378939004850176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378939004850176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168284281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22469370ØxzÙ'ÓO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372568891041792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93978096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93968119 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45648008 trainer/global_step‚9899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384780411141120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22469126 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6034869897216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843727100126963e+09 _step‚801 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74306183 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384780516117504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5841494227968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6034695690240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372568874002432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168274302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5841406290944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93978096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74306185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74306183 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45648005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚378389733164143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚378939004850176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378939004850176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168284281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22469370 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5820945298801 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378939021889536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5841494227968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74306185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356613663207424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362648558008320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5820859161711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68117131 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384780411141120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚378939021889536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362648358897664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144¡/êµã2‘- _wandbruntime‚4651  _runtime‚4651+Úß5"’ØM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168300797 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168290804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384818196448256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93987391 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚378976115814400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5841975650304 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362684058502144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362683863592960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68123994 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6035288870912 _runtime‚4651.791699806 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93987391 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93977400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168290804 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74313404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22471275 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68123764 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚802 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚9899 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168300797 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74313404 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843727103476644e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45652489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378426772521001 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5842063594496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6035458876416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5821512620443 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356648574722048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚356648599625728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22471502 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064eýFçã&“ÝM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168290804 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74313404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22471275 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68123764 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _step‚802 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚9899 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168300797 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74313404 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843727103476644e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45652489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378426772521001 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5842063594496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6035458876416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5821512620443 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356648574722048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚356648599625728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22471502 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168300797 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168290804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384818196448256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93987391 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚378976115814400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5841975650304 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362684058502144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362683863592960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68123994 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6035288870912 _runtime‚4651.791699806 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93987391 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93977400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936¢³s<2”- _wandbruntime‚4651  _runtime‚4651u¬& •„ %loss/total_loss‚7.981601530385833 _runtime‚4651.793106783 /loss/FRt2/assignment_loss‚0.644498888689738 1loss/SRqqt2/assignment_loss‚0.487261609887479 1loss/SRqqt1/detection_loss‚0.7483334766950235 0loss/FBt1/detection_loss‚0.49611849065870045 % _timestamp‚1.7843727103495734e+09 _step‚803 3loss/FBt1/assignment_loss‚0.0005974460294601158 /loss/FRt1/detection_loss‚0.5992743101787675 1loss/SRqqt2/detection_loss‚0.6854404826458036 trainer/global_step‚9899 epoch‚12 0loss/FRt1/assignment_loss‚1.5648782188179848 2loss/SRqqt1/assignment_loss‚0.5791848451285188 2loss/FBt2/assignment_loss‚0.034830544144893914 /loss/FRt2/detection_loss‚1.4428533174482605 /loss/FBt2/detection_loss‚0.6983299000612037ÖÊ–‰ /loss/FBt2/detection_loss‚0.6983299000612037 %loss/total_loss‚7.981601530385833 _runtime‚4651.793106783 /loss/FRt2/assignment_loss‚0.644498888689738 1loss/SRqqt2/assignment_loss‚0.487261609887479 1loss/SRqqt1/detection_loss‚0.7483334766950235 0loss/FBt1/detection_loss‚0.49611849065870045 % _timestamp‚1.7843727103495734e+09 _step‚803 3loss/FBt1/assignment_loss‚0.0005974460294601158 /loss/FRt1/detection_loss‚0.5992743101787675 1loss/SRqqt2/detection_loss‚0.6854404826458036 trainer/global_step‚9899 epoch‚12 0loss/FRt1/assignment_loss‚1.5648782188179848 2loss/SRqqt1/assignment_loss‚0.5791848451285188 2loss/FBt2/assignment_loss‚0.034830544144893914 /loss/FRt2/detection_loss‚1.4428533174482605£½é}Æ‘—‚(:† ó»íÒ¯ÝÞ‹ gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚188.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ŒK똂(:à ó»íÒù­ž disk.rbd1.out‚ 11.08203125 disk.sda1.in‚0 disk.dm-1.in‚0.109375# disk.dm-0.out‚13200.40087890625 disk.sda.in‚0.109375) disk./.usagePercent‚39.87601258148003 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 144.69921875# disk.nvme0c0n1.in‚ 6310.16796875$ memory_percent‚99.09775257110596* proc.memory.percent‚16.331791877746582 cpu‚26.86612918886141 disk.rbd0.out‚0& disk.nvme0n1.out‚13200.40087890625( proc.memory.availableMB‚ 147.82421875# proc.memory.rssMB‚ 2675.80078125! disk.nvme0n1.in‚ 6310.16796875 network.sent‚ 1.3424196e+07 network.recv‚ 1.452866e+06 disk.sda.out‚ 144.69921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0.109375 disk.sda3.out‚ 144.69921875$ disk./.usageGB‚594.1814041137695( disk.nvme0c0n1.out‚13200.40087890625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6310.16796875nó`°2™- _wandbruntime‚4673  _runtime‚4673)DÆšÀ trainer/global_step‚9949 % _timestamp‚1.7843727316849456e+09 _runtime‚ 4673.12876683 _step‚804 & lr-AdamW/pg1‚0.0007393756005454524 & lr-AdamW/pg2‚0.0007393756005454524Á Bx˛Š_runtime‚ 4673.12876683 _step‚804 & lr-AdamW/pg1‚0.0007393756005454524 & lr-AdamW/pg2‚0.0007393756005454524 trainer/global_step‚9949 % _timestamp‚1.7843727316849456e+09¤òjËo2œ- _wandbruntime‚4673  _runtime‚4673.Öß_Ô'ÎO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380794570100736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364423608632320 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380794553061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5869994452992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74667235 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358359263842304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚9949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94434021 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380794570100736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45872205 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74667235 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5869906515968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169111233 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380241836086701 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386664459577344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386664459577344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380241732910251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386664564553728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5869994452992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5869906515968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358359238938624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68450360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚374392492808192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94443998 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6064344790016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849343278509 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 _step‚805 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22578155 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94443998 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚374392475768832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74667233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4673.130026358 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22578155 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94443998 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚374392475768832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74667233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4673.130026358 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380794553061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5869994452992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74667235 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358359263842304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚9949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94434021 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380794570100736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45872205 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74667235 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5869906515968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169111233 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380241836086701 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386664459577344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386664459577344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380241732910251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386664564553728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5869994452992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5869906515968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358359238938624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68450360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚374392492808192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚“<.âP 327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94443998 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6064344790016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849343278509 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 _step‚805 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358394150453248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380831664025600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22580465 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843727320255284e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386702244884480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386702139900928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6064938323968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380831664025600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386702244884480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364458912480256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚374428964843520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94453293 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68456942 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5849824462949 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380278875443671 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68457157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5870563819520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74674454 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380831681064960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74674454 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169127749 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5870475875328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45876692 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169127749 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169117756 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚215 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22580253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94443302 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380278772267109 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169117756 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5870563819520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380831681064960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144ñPˆã&¡ÝM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380278875443671 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68457157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5870563819520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74674454 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380831681064960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74674454 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169127749 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5870475875328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45876692 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169127749 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169117756 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚215 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22580253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94443302 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380278772267109 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169117756 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5870563819520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380831681064960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358394150453248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380831664025600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22580465 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843727320255284e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386702244884480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386702139900928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6064938323968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380831664025600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386702244884480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364458912480256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚374428964843520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94453293 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68456942 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5849824462949 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¦‹¯)2¢- _wandbruntime‚4673  _runtime‚4673o £‡ epoch‚12 % _timestamp‚1.7843727320274174e+09 _step‚807 0loss/FRt2/assignment_loss‚0.8228309490148303 /loss/FBt1/detection_loss‚0.3374857124355104 trainer/global_step‚9949 %loss/total_loss‚7.988756047514588 _runtime‚4673.470950761 2loss/SRqqt1/assignment_loss‚0.4804059396804299 2loss/SRqqt2/assignment_loss‚0.5717207597208926 4loss/FBt1/assignment_loss‚0.00020931976292912522 1loss/SRqqt1/detection_loss‚1.0173701397114736 0loss/FBt2/detection_loss‚0.39059216192714413 /loss/FRt1/assignment_loss‚1.653752973344591 3loss/FBt2/assignment_loss‚1.682272535109122e-05 /loss/FRt1/detection_loss‚0.5309249264081262 /loss/FRt2/detection_loss‚1.5298511867673652 1loss/SRqqt2/detection_loss‚0.6535951560159446³ª”¤Œ 1loss/SRqqt1/detection_loss‚1.0173701397114736 0loss/FBt2/detection_loss‚0.39059216192714413 /loss/FRt1/assignment_loss‚1.653752973344591 3loss/FBt2/assignment_loss‚1.682272535109122e-05 /loss/FRt1/detection_loss‚0.5309249264081262 /loss/FRt2/detection_loss‚1.5298511867673652 1loss/SRqqt2/detection_loss‚0.6535951560159446 epoch‚12 % _timestamp‚1.7843727320274174e+09 _step‚807 0loss/FRt2/assignment_loss‚0.8228309490148303 /loss/FBt1/detection_loss‚0.3374857124355104 trainer/global_step‚9949 %loss/total_loss‚7.988756047514588 _runtime‚4673.470950761 2loss/SRqqt1/assignment_loss‚0.4804059396804299 2loss/SRqqt2/assignment_loss‚0.5717207597208926 4loss/FBt1/assignment_loss‚0.00020931976292912522§Ômú±›¥‚(: ‚¼íÒâ— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚183.984' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.796800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚g–ôݦ‚(:Ò ‚¼íÒ·ƒðŽ disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2675.80078125 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 146.15625 disk.dm-0.in‚ 6310.16796875$ disk./.usageGB‚594.1541442871094 disk.sda3.out‚ 146.15625& disk.nvme0c0n1.out‚13213.822265625! disk.nvme0n1.in‚ 6310.16796875 disk.rbd0.out‚0 disk.sda3.in‚0.109375 disk.rbd1.out‚ 11.14453125 cpu‚26.864146432135083* disk./.usagePercent‚39.874183151640345 disk.sda.out‚ 146.15625 disk.sda1.in‚0 network.sent‚ 1.3486728e+07 disk.sda.in‚0.109375* proc.memory.percent‚16.331791877746582 disk.rbd0.in‚0! disk.dm-0.out‚13213.822265625$ memory_percent‚99.09787178039551' proc.memory.availableMB‚ 147.8046875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6310.16796875 disk.sda1.out‚0 disk.dm-1.in‚0.109375 proc.cpu.threads‚17 network.recv‚ 1.457527e+06$ disk.nvme0n1.out‚13213.822265625§IÄꑧ‚(:† ‘¼íÒÿ伊 gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷³[i娂(:Ú ‘¼íÒ¸³ÁŒ disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 146.66796875" disk.dm-0.out‚13244.2919921875 disk.sda3.in‚0.109375" disk.nvme0c0n1.in‚ 6410.9609375 disk.rbd0.out‚0 network.sent‚ 1.3549574e+07) disk./.usagePercent‚39.82446418442308 disk.sda.in‚0.109375' disk.nvme0c0n1.out‚13244.2919921875' proc.memory.availableMB‚ 147.7734375 network.recv‚ 1.462543e+06 disk.sda1.out‚0 disk.sda3.out‚ 146.66796875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2675.80078125$ disk./.usageGB‚593.4132957458496 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6410.9609375* proc.memory.percent‚16.331791877746582 proc.cpu.threads‚17 disk.dm-1.in‚0.109375 disk.rbd1.in‚0% disk.nvme0n1.out‚13244.2919921875 disk.sda.out‚ 146.66796875 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 6410.9609375 disk.rbd1.out‚ 11.14453125$ memory_percent‚99.09806251525879 cpu‚26.862170902590737vˆÝâ2©- _wandbruntime‚4695  _runtime‚4695‡2ÎOƪÀ _step‚808 & lr-AdamW/pg1‚0.0007382385666909938 & lr-AdamW/pg2‚0.0007382385666909938 trainer/global_step‚9999 % _timestamp‚1.7843727535647492e+09 _runtime‚ 4695.00851672Þ-þ†Ë«Å & lr-AdamW/pg1‚0.0007382385666909938 & lr-AdamW/pg2‚0.0007382385666909938 trainer/global_step‚9999 % _timestamp‚1.7843727535647492e+09 _runtime‚ 4695.00851672 _step‚808¨êv=2¬- _wandbruntime‚4695  _runtime‚4695í y­ÌO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75028283 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366198434525696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6093619855872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388548612989952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4695.009710423 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169928206 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94909900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46096405 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366198642024960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382650118311936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚382650118311936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 trainer/global_step‚9999 _step‚809 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388548612989952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5877741258217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388548508013568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388548508013568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 30618CHs¹4192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94909900 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94899923 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360104814669824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382650101272576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75028283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ãwW×'®ÑO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388548508013568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388548508013568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94909900 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94899923 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360104814669824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382650101272576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75028283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75028283 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366198434525696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6093619855872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388548612989952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4695.009710423 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169928206 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94909900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46096405 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366198642024960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382650118311936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚382650118311936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 trainer/global_step‚9999 _step‚809 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388548612989952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5877741258217©žfïw2¯- _wandbruntime‚4695  _runtime‚4695îWÂÝ&°×M EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366234139295232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382687229276160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5878308579859 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169944708 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382687212236800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382687229276160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68790702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _step‚810 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5899064044544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382130772013217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22689813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366233931803136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 $ _timestamp‚1.784372753904102e+09 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388586293320704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94919195 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46100892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360139726184448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6094388207104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376252566609920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382130875189779 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5899064044544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382687212236800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46100889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68790942 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22690050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388586188337152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94909204 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169954701 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75035504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94909204 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚9999 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75035504 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4695.348178485 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75035506 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376252549570560 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75035506 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5878222442657 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169954701 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169944708 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94919195 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388586293320704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064ªÌ`y>±ÜM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94909204 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169954701 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75035504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94909204 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 trainer/global_step‚9999 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75035504 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4695.348178485 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75035506 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376252549570560 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75035506 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5878222442657 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169954701 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169944708 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94919195 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388586293320704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366234139295232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382687229276160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5878308579859 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169944708 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382687212236800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382687229276160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68790702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _step‚810 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5899064044544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382130772013217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22689813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366233931803136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 $ _timestamp‚1.784372753904102e+09 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388586293320704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94919195 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46100892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360139726184448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6094388207104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376252566609920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JꦤE?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382130875189779 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5899064044544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382687212236800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46100889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68790942 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22690050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388586188337152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2ªaN‘2²- _wandbruntime‚4695  _runtime‚4695Š3)u ³‡ trainer/global_step‚9999 % _timestamp‚1.7843727539057896e+09 0loss/FBt2/detection_loss‚0.48492987784009206 0loss/FRt1/assignment_loss‚1.6687629078626633 /loss/FRt2/detection_loss‚1.4560148629373142 1loss/SRqqt2/detection_loss‚0.6250275873647231 _runtime‚4695.350837766 _step‚811 2loss/FBt2/assignment_loss‚0.005533599780532479 epoch‚12 0loss/FRt2/assignment_loss‚0.5596439844059153 3loss/SRqqt2/assignment_loss‚0.45843688105245434 3loss/FBt1/assignment_loss‚6.810951422724859e-05 /loss/FRt1/detection_loss‚0.5467746448143007 1loss/SRqqt1/detection_loss‚0.8180169225294397 2loss/SRqqt1/assignment_loss‚0.7199275915650868 /loss/FBt1/detection_loss‚0.5529608607154202 %loss/total_loss‚7.896097830382169 Xå~´Œ 2loss/FBt2/assignment_loss‚0.005533599780532479 epoch‚12 0loss/FRt2/assignment_loss‚0.5596439844059153 3loss/SRqqt2/assignment_loss‚0.45843688105245434 3loss/FBt1/assignment_loss‚6.810951422724859e-05 /loss/FRt1/detection_loss‚0.5467746448143007 1loss/SRqqt1/detection_loss‚0.8180169225294397 2loss/SRqqt1/assignment_loss‚0.7199275915650868 /loss/FBt1/detection_loss‚0.5529608607154202 %loss/total_loss‚7.896097830382169 trainer/global_step‚9999 % _timestamp‚1.7843727539057896e+09 0loss/FBt2/detection_loss‚0.48492987784009206 0loss/FRt1/assignment_loss‚1.6687629078626633 /loss/FRt2/detection_loss‚1.4560148629373142 1loss/SRqqt2/detection_loss‚0.6250275873647231 _runtime‚4695.350837766 _step‚811« É›(‘µ‚(:†  ¼íÒ˜€²Š gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚180.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å‹Aßܶ‚(:Ñ  ¼íÒ˜²åŒ% disk.nvme0n1.out‚13251.5341796875 disk.sda2.out‚0 disk.dm-1.in‚0.109375 network.sent‚ 1.3555071e+07 network.recv‚ 1.465701e+06 disk.dm-1.out‚ 147.46875 disk.sda.in‚0.109375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚13251.5341796875 disk.rbd0.in‚0# memory_percent‚99.0994930267334# proc.memory.rssMB‚ 2675.80078125* disk./.usagePercent‚39.822572032737504 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.331791877746582 proc.cpu.threads‚17 disk.sda3.in‚0.109375 disk.sda3.out‚ 147.46875 disk.drbd2165.out‚0 cpu‚26.860282102270762 disk.sda.out‚ 147.46875 disk.dm-0.in‚ 6410.9609375' proc.memory.availableMB‚ 147.5390625" disk.nvme0c0n1.in‚ 6410.9609375 disk.nvme0n1.in‚ 6410.9609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚13251.5341796875$ disk./.usageGB‚593.3851013183594 disk.rbd1.out‚ 11.20703125FÖƒ2·- _wandbruntime‚4716  _runtime‚4716R¶cǸÁ & lr-AdamW/pg2‚0.0007370965001543266 trainer/global_step‚10049 % _timestamp‚1.7843727751822705e+09 _runtime‚ 4716.62766891 _step‚812 & lr-AdamW/pg1‚0.0007370965001543266Û€uÌ¹Æ % _timestamp‚1.7843727751822705e+09 _runtime‚ 4716.62766891 _step‚812 & lr-AdamW/pg1‚0.0007370965001543266 & lr-AdamW/pg2‚0.0007370965001543266 trainer/global_step‚10049¬†²šý2º- _wandbruntime‚4716  _runtime‚4716œætŽÕ'»ÏO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361850390401024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390432556449792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5926994903040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75389335 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5906053100835 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383945835578917 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5926994903040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5926906966016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46320605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384505666523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75389335 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367973686585856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378039696340992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378039679301632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69117254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367973483280896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6123092879872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95375802 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384505649483776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75389333 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22796649 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚10049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95365825 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6123271281152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22796899 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843727751830282e+09 _runtime‚4716.627754632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5926906966016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95365825 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46320608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390432556449792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5926994903040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75389335 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5906053100835 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383945835578917 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5926994903040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5926906966016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46320605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384505666523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75389335 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367973686585856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378039696340992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378039679301632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69117254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367973483280896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6123092879872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95375802 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384505649483776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75389333 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22796649 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚10049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95365825 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6123271281152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22796899 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843727751830282e+09 _runtime‚4716.627754632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5926906966016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95365825 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46320608 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46325089 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383982771759325 _step‚814 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5927564269568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5927476325376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5906620422365 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170771660 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75396556 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚10049 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95375106 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95385097 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95375106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384542777487360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46325092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22799084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22798851 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378076151336960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170781653 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384542777487360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368008986522112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390470341756928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6123860903424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361885326819328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383982874935887 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75396554 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95385097 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368009187722752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4716.962679744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69123940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5927476325376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361885301915648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378076168376320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192Êsä&¿ÞM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5927476325376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5906620422365 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170771660 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75396556 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚10049 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95375106 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95385097 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95375106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384542777487360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46325092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22799084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22798851 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378076151336960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170781653 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384542777487360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368008986522112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390470341756928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6123860903424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361885326819328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383982874935887 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75396554 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95385097 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368009187722752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4716.962679744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69123940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5927476325376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361885301915648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378076168376320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5906706559567 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170771660 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75396554 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69124176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46325089 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383982771759325 _step‚814 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5927564269568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0®žaä2À- _wandbruntime‚4716  _runtime‚4716´šª Á† /loss/FRt2/assignment_loss‚0.653838784784399 3loss/SRqqt2/assignment_loss‚0.47456599805482635 /loss/FRt1/detection_loss‚0.5736477726862352 .loss/FRt2/detection_loss‚1.595951026465784 0loss/FRt1/assignment_loss‚1.5632008220079734 1loss/SRqqt2/detection_loss‚0.5988772474060005 /loss/FBt1/detection_loss‚0.8105937663465739 /loss/FBt2/detection_loss‚0.4371530592971895 _runtime‚4716.964066422 2loss/SRqqt1/assignment_loss‚0.6241186126111217 3loss/FBt2/assignment_loss‚0.0002723109945850726 1loss/SRqqt1/detection_loss‚0.8456222365194627 epoch‚12 trainer/global_step‚10049 % _timestamp‚1.7843727755205252e+09 _step‚815 3loss/FBt1/assignment_loss‚4.722455353025623e-05 %loss/total_loss‚8.177888861727682ž‡fá‹ 0loss/FRt1/assignment_loss‚1.5632008220079734 1loss/SRqqt2/detection_loss‚0.5988772474060005 /loss/FBt1/detection_loss‚0.8105937663465739 /loss/FBt2/detection_loss‚0.4371530592971895 _runtime‚4716.964066422 2loss/SRqqt1/assignment_loss‚0.6241186126111217 3loss/FBt2/assignment_loss‚0.0002723109945850726 1loss/SRqqt1/detection_loss‚0.8456222365194627 epoch‚12 trainer/global_step‚10049 % _timestamp‚1.7843727755205252e+09 _step‚815 3loss/FBt1/assignment_loss‚4.722455353025623e-05 %loss/total_loss‚8.177888861727682 /loss/FRt2/assignment_loss‚0.653838784784399 3loss/SRqqt2/assignment_loss‚0.47456599805482635 /loss/FRt1/detection_loss‚0.5736477726862352 .loss/FRt2/detection_loss‚1.595951026465784¯ð’ ð‘Â(:† ¯¼íÒß­’Œ gpu.0.gpu‚61.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒŽqqâÄ‚(:× ¯¼íÒÙϹ network.recv‚ 1.470418e+06$ disk.nvme0n1.out‚13256.888671875 disk.sda1.in‚0 disk.sda3.in‚0.109375" disk.nvme0c0n1.in‚ 6410.9609375& disk.nvme0c0n1.out‚13256.888671875 network.sent‚ 1.3617935e+07 disk.nvme0n1.in‚ 6410.9609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 147.66015625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2675.80078125 disk.sda2.in‚0 disk.sda.out‚ 147.66015625* proc.memory.percent‚16.331791877746582 proc.cpu.threads‚17$ disk./.usageGB‚593.3852348327637 disk.dm-1.in‚0.109375 disk.rbd1.out‚ 11.20703125$ memory_percent‚99.10004138946533 cpu‚26.858295633277194! disk.dm-0.out‚13256.888671875( disk./.usagePercent‚39.8225809930012 disk.dm-1.out‚ 147.66015625 disk.dm-0.in‚ 6410.9609375 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda.in‚0.109375 disk.rbd1.in‚0( proc.memory.availableMB‚ 147.44921875 x2Å- _wandbruntime‚4738  _runtime‚4738yÁ5™ÈÆ & lr-AdamW/pg2‚0.0007359494192279643 trainer/global_step‚10099 % _timestamp‚1.7843727971469955e+09 _runtime‚4738.590802069 _step‚816 & lr-AdamW/pg1‚0.0007359494192279643¥Ó"ÍÇÇ _runtime‚4738.590802069 _step‚816 & lr-AdamW/pg1‚0.0007359494192279643 & lr-AdamW/pg2‚0.0007359494192279643 trainer/global_step‚10099 % _timestamp‚1.7843727971469955e+09°ö4m2È- _wandbruntime‚4738  _runtime‚4738ã6 Õ'ÉÏO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386361197694976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚379863281068032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚817 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22906191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369748568272384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95841704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22905969 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5934451080543 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392316604886016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369748761091584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363595966132224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69450999 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69450774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46544805 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75750383 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95831727 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392316604886016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386361197694976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6152770055680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5955407191040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚4738.592016141 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95831727 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386361214734336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171592089 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171582110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95841704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386361214734336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392316709862400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚379863298107392 % _timestamp‚1.7843727971480417e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5955407191040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363595991035904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392316709862400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5955495128064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934537217633 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5955495128064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75750385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75750385 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75750383 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46544808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚385797835325025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327i‹v¶ÊÔO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5955407191040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363595991035904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392316709862400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5955495128064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934537217633 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5955495128064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75750385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75750385 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75750383 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46544808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚385797835325025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386361197694976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚379863281068032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚817 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22906191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369748568272384 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95841704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22905969 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5934451080543 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3923166048>SÞ«$86016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369748761091584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363595966132224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69450999 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69450774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46544805 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75750383 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95831727 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392316604886016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167915520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386361197694976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6152770055680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5955407191040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 _runtime‚4738.592016141 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95831727 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386361214734336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171592089 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171582110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95841704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386361214734336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392316709862400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚379863298107392 % _timestamp‚1.7843727971480417e+09±û­'2Ë- _wandbruntime‚4738  _runtime‚4738á—Æ ß&ÌÙM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171608605 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386398325698560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5955976550400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379899753103360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95850999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _step‚818 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171598612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369784065791488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69457686 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5955976550400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171608605 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75757604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4738.934140988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379899770142720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46549289 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22908167 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69457456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75757604 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386398308659200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5956064494592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363630902550528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95850999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4738.934140988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379899770142720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46549289 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22908167 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69457456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75757604 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386398308659200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5956064494592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363630902550528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95850999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386398325698560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5955976550400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379899753103360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95850999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 _step‚818 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171598612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369784065791488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69457686 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5955976550400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171608605 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75757604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76111435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23014924 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96307606 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69783929 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371523605170176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 $ _timestamp‚1.784372818967285e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚388216745906176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5983995353088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387649835071133 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388216745906176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172419041 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394200653322240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96297629 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394200758298624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388216762945536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5983907416064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76111433 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381686882834432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76111433 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69784168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172409062 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96297629 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚365341566767104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394200653322240ðHD¥Ù'ÚÓO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394200653322240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚388216762945536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚365341541863424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76111435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23014924 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96307606 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69783929 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371523605170176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 $ _timestamp‚1.784372818967285e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚388216745906176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5983995353088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387649835071133 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388216745906176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172419041 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394200653322240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96297629 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394200758298624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388216762945536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5983907416064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76111433 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381686882834432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76111433 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69784168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172409062 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96297629 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚365341566767104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0µfpFÜ2Û- _wandbruntime‚4760  _runtime‚4760@©ÀÞ&ÜØM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5963416381781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76118656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381723354869760 trainer/global_step‚10149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388253873909760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5984564719616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5984476775424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388253873909760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46773489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371559099752448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚365376478281728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387686874428103 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 $ _timestamp‚1.784372819309673e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387686771251541 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172435557 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6182828962816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394238438629376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46773492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23017306 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76118656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚365376453378048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172425564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5984476775424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚388253856870400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371559307244544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4760.753757146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23017079 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76118654 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5963502518983 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69790568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96306910 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394238333645824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394238438629376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69790798 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6182646374400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96316901 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96306910 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172425564“äðQÝÝM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394238333645824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394238438629376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69790798 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6182646374400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96316901 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96306910 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172425564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5963416381781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76118656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381723354869760 trainer/global_step‚10149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388253873909760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5984564719616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5984476775424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388253873909760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46773489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371559099752448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚365376478281728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387686874428103 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 $ _timestamp‚1.784372819309673e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207492096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387686771251541 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172435557 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6182828962816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182588416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMoâý’nitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394238438629376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46773492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23017306 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76118656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚365376453378048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172425564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5984476775424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚388253856870400 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371559307244544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4760.753757146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23017079 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76118654 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5963502518983 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69790568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96306910 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396084701758464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390072294117376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5991333177049 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96773508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390072311156736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚4782.335696221 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389501834817241 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96763531 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396084806734848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390072294117376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6012495578112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23123901 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46993205 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373298834252800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6211691754496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76472485 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚396084806734848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5991247039959 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843728408916914e+09 _step‚825 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173236014 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6012495578112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76472483 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96773508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚383510501640192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6211515450368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0˜n DÚ'èÔO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76472483 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396084701758464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390072311156736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚4782.335696221 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389501834817241 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96763531 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396084806734848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390072294117376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6012495578112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23123901 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46993205 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373298834252800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6211691754496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76472485 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚396084806734848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5991247039959 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843728408916914e+09 _step‚825 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173236014 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6012495578112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76472483 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96773508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚383510501640192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6211515450368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390072311156736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6012407641088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396084701758464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390072294117376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5991333177049 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96773508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76479704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6013064944640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70123555 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96772812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173262509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70123794 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6012977000448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76479704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389538874174211 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚383546973675520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚10199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390109405081600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6012977000448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDevicåýæ veStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5991900498691 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96782803 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373334332950528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 % _timestamp‚1.7843728412365384e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚396122487065600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390109422120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76479706 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46997689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23126102 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383546956636160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46997692 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367122054012928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6212108932096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96772812 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6212278937600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚826 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5991814361489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚396122382082048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173252516 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373334138041344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367122029109248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173262509 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76479706 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4782.680674011 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390109422120960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70123555 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96772812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173262509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70123794 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6012977000448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76479704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389538874174211 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚383546973675520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 trainer/global_step‚10199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390109405081600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6012977000448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5991900498691 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96782803 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373334332950528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 % _timestamp‚1.7843728412365384e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚396122487065600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390109422120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76479706 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46997689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23126102 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383546956636160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46997692 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367122054012928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6212108932096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96772812 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6212278937600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚826 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5991814361489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚396122382082048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173252516 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373334138041344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367122029109248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173262509 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76479706 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4782.680674011 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390109422120960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144ºÅåÝ2ì- _wandbruntime‚4782  _runtime‚4782 }áuí‚ $loss/total_loss‚8.31474096823405 trainer/global_step‚10199 0loss/FRt1/assignment_loss‚1.6403951054421542 1loss/FBt1/assignment_loss‚0.00677036525913892 1loss/SRqqt1/detection_loss‚0.8231988816643768 0loss/FBt2/detection_loss‚0.37878223464054905 % _timestamp‚1.7843728412383347e+09 _step‚827 0loss/FRt2/assignment_loss‚0.7807970410781288 2loss/SRqqt1/assignment_loss‚0.6540863049558125 1loss/FBt2/assignment_loss‚0.06438125956374048 .loss/FRt2/detection_loss‚1.394637005158783 /loss/FBt1/detection_loss‚0.8690928550259698 _runtime‚4782.681867474 1loss/SRqqt2/detection_loss‚0.6082149640299326 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5104799719392864 /loss/FRt1/detection_loss‚0.5839049794761765?nMù î‡ 0loss/FRt1/assignment_loss‚1.6403951054421542 1loss/FBt1/assignment_loss‚0.00677036525913892 1loss/SRqqt1/detection_loss‚0.8231988816643768 0loss/FBt2/detection_loss‚0.37878223464054905 % _timestamp‚1.7843728412383347e+09 _step‚827 0loss/FRt2/assignment_loss‚0.7807970410781288 2loss/SRqqt1/assignment_loss‚0.6540863049558125 1loss/FBt2/assignment_loss‚0.06438125956374048 .loss/FRt2/detection_loss‚1.394637005158783 /loss/FBt1/detection_loss‚0.8690928550259698 _runtime‚4782.681867474 1loss/SRqqt2/detection_loss‚0.6082149640299326 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5104799719392864 /loss/FRt1/detection_loss‚0.5839049794761765 $loss/total_loss‚8.31474096823405 trainer/global_step‚10199»‚?õØœï‚(:‘ ë¼íÒϤڋ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚120.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.080399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dACéð‚(:Þ ë¼íÒ¤ª– disk.drbd2165.out‚0% disk.nvme0n1.out‚13300.1982421875 network.recv‚ 1.486173e+06 disk.sda1.in‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0* disk./.usagePercent‚39.822693380308735 disk.sda1.out‚0$ disk./.usageGB‚593.3869094848633 disk.sda2.in‚0 disk.rbd1.out‚ 11.33203125# disk.nvme0c0n1.in‚ 6501.01953125( proc.memory.availableMB‚ 147.70703125 disk.rbd0.out‚0! disk.nvme0n1.in‚ 6501.01953125# proc.memory.rssMB‚ 2675.80078125* proc.memory.percent‚16.331791877746582 network.sent‚ 1.3810259e+07 disk.rbd0.in‚0 disk.sda3.in‚0.109375 disk.sda.out‚ 153.92578125 cpu‚26.85076735418014 disk.dm-1.in‚0.109375" disk.dm-0.out‚13300.1982421875$ memory_percent‚99.09846782684326 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚ 6501.01953125 disk.sda3.out‚ 153.92578125 disk.rbd1.in‚0 disk.dm-1.out‚ 153.92578125' disk.nvme0c0n1.out‚13300.1982421875ìeØ®‘ñ‚(:† ú¼íҧЌ gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚157.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.5504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯UYóÛò‚(:Ð ú¼íÒÎ𜎠disk.dm-1.in‚0.109375 cpu‚26.84888836568272$ disk./.usageGB‚593.4006118774414 disk.sda3.out‚ 157.4453125# disk.nvme0c0n1.in‚ 6501.01953125 disk.sda2.in‚0 disk.dm-0.in‚ 6501.01953125 proc.cpu.threads‚17 network.recv‚ 1.490338e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.331791877746582 network.sent‚ 1.3872798e+07 disk.rbd1.out‚ 11.45703125$ disk.nvme0c0n1.out‚ 13329.1484375) disk./.usagePercent‚39.82361295937173 disk.rbd1.in‚0! disk.nvme0n1.in‚ 6501.01953125# proc.memory.rssMB‚ 2675.80078125' proc.memory.availableMB‚ 147.7109375 disk.sda3.in‚0.109375 disk.rbd0.in‚0$ memory_percent‚99.09844398498535 disk.sda.out‚ 157.4453125 disk.drbd2165.in‚0" disk.nvme0n1.out‚ 13329.1484375 disk.dm-1.out‚ 157.4453125 disk.sda.in‚0.109375 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 13329.1484375¾qŽ©2ó- _wandbruntime‚4804  _runtime‚4804©®ýºÆôÀ _runtime‚4804.067253812 _step‚828 % lr-AdamW/pg1‚0.000732478274238812 % lr-AdamW/pg2‚0.000732478274238812 trainer/global_step‚10249 % _timestamp‚1.7843728626234574e+09., ËõÅ % _timestamp‚1.7843728626234574e+09 _runtime‚4804.067253812 _step‚828 % lr-AdamW/pg1‚0.000732478274238812 % lr-AdamW/pg2‚0.000732478274238812 trainer/global_step‚10249¼"è%v2ö- _wandbruntime‚4804  _runtime‚4804Üm¢mÕ'÷ÏO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375073846880256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 % _timestamp‚1.7843728626245213e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6240946055168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚391353834563349 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6241128650752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23232884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70450289 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23233125 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76833535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397968750194688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397968855171072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚391927859367936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397968855171072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174072945 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97229433 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚391927859367936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385334086367232 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397968750194688 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6040995803136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚391353731386899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368832693325824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚4804.068585316ü®¼é;øÔO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397968750194688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397968855171072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚391927859367936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397968855171072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.cur¨cŸ rent‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174072945 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97229433 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚391927859367936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385334086367232 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397968750194688 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6040995803136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚391353731386899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368832693325824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚4804.068585316 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375073846880256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 % _timestamp‚1.7843728626245213e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6240946055168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚391353834563349 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6241128650752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23232884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70450289 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23233125 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76833535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146½ÇD¨Í2ù- _wandbruntime‚4804  _runtime‚4804Vâpnß&úÙM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174079468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174089461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76840756 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375109341737984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23235095 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368867604840448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6241531502592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385370575441920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚391964953292800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97238714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6041477225472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174089461 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6241711993856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398006535501824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174079468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6020212341197 trainer/global_step‚10249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47221889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398006535501824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70456984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚830 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76840754 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚391964970332160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6041565169664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76840756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70457210 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97248705 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97248705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23235318 % _timestamp‚1.7843728629653707e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97238714 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368867629744128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0šÝ±•ä&ûÞM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398006535501824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174079468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6020212341197 trainer/global_step‚10249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47221889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398006535501824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70456984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚830 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76840754 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚391964970332160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6041565169664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76840756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205394944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70457210 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97248705 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97248705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23235318 % _timestamp‚1.7843728629653707e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97238714 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368867629744128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174079468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174089461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76840756 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375109341737984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23235095 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368867604840448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6241531502592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385370575441920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚391964953292800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97238714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6041477225472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174089461 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6241711993856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399852798630912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77194585 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23341569 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMon¡yãÙ€$itor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270399803392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370578269057024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387157705172992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77194583 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393783407579136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387157688133632 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376848865873920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6048129136465 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97705312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6270571913216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399852798630912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6069496028160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393783390539776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97695335 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399852903607296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393783390539776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97705312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6069496028160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47441608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47441605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23341809 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70783417 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6069408091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4826.008584369 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376848668860416 % _timestamp‚1.7843728845610366e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70783174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174899897 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97695335 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399852903607296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393783407579136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _step‚833 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174889918 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399852798630912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77194585 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23341569 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270399803392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370578269057024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387157705172992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77194583 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393783407579136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387157688133632 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376848865873920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6048129136465 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97705312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6270571913216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399852798630912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6069496028160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393783390539776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97695335 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399852903607296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393783390539776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97705312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6069496028160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47441608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47441605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23341809 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70783417 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6069408091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4826.008584369 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376848668860416 % _timestamp‚1.7843728845610366e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70783174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174899897 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97695335 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399852903607296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393783407579136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _step‚833 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174889918 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393820518543360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6070065394688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174916413 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271160382464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393242770489865 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843728848993418e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174906420 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97704616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23343949 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23343721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387194160168960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚834 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4826.343489084 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77201804 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6069977450496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚10299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77201804 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70790041 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47446092 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚393820501504000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370613205475328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6270990376960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393242873666427 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399890583938048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6048610320905 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376884365857792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97714607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399890583938048 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70789810 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387194177208320‹Àor ‰ÞM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271160382464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393242770489865 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843728848993418e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174906420 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97704616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23343949 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23343721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387194160168960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚834 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4826.343489084 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77201804 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6069977450496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚10299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77201804 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70790041 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allo¦írcated‚47446092 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚393820501504000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370613205475328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6270990376960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393242873666427 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399890583938048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6048610320905 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376884365857792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97714607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399890583938048 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70789810 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387194177208320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370613180571648 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393820518543360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6070065394688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174916413 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128ÂÁ²o52Š- _wandbruntime‚4826  _runtime‚4826Ø4 ‹„ trainer/global_step‚10299 0loss/FRt1/assignment_loss‚1.6080648169009644 0loss/FRt2/assignment_loss‚0.5751396782544195 1loss/SRqqt2/assignment_loss‚0.543747590668278 /loss/FBt2/detection_loss‚0.4733795908654833 _runtime‚4826.344555269 _step‚835 2loss/SRqqt1/assignment_loss‚0.5897155435785641 3loss/FBt1/assignment_loss‚0.0007444398792885853 1loss/FBt2/assignment_loss‚0.01120432530117028 1loss/SRqqt2/detection_loss‚0.6203067028206651 /loss/FBt1/detection_loss‚0.5198424510835182 %loss/total_loss‚7.788587676480092 /loss/FRt1/detection_loss‚0.5158888611361343 1loss/SRqqt1/detection_loss‚0.7427700317971762 epoch‚12 % _timestamp‚1.7843728849010007e+09 /loss/FRt2/detection_loss‚1.5877836441944295†…èJŒ‰ 1loss/SRqqt2/assignment_loss‚0.543747590668278 /loss/FBt2/detection_loss‚0.4733795908654833 _runtime‚4826.344555269 _step‚835 2loss/SRqqt1/assignment_loss‚0.5897155435785641 3loss/FBt1/assignment_loss‚0.0007444398792885853 1loss/FBt2/assignment_loss‚0.01120432530117028 1loss/SRqqt2/detection_loss‚0.6203067028206651 /loss/FBt1/detection_loss‚0.5198424510835182 %loss/total_loss‚7.788587676480092 /loss/FRt1/detection_loss‚0.5158888611361343 1loss/SRqqt1/detection_loss‚0.7427700317971762 epoch‚12 % _timestamp‚1.7843728849010007e+09 /loss/FRt2/detection_loss‚1.5877836441944295 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚1.6080648169009644 0loss/FRt2/assignment_loss‚0.5751396782544195Ã*:l‘‚(:† ˜½íÒÚ…Š gpu.0.gpu‚47.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-©sï玂(:Ü ˜½íÒí¢£Œ# proc.memory.rssMB‚ 2675.80078125( disk.nvme0c0n1.out‚13341.07861328125 disk.sda1.out‚0 disk.dm-0.in‚ 6611.1484375 disk.drbd2165.in‚0# memory_percent‚99.1020679473877 disk.rbd1.in‚0 disk.nvme0n1.in‚ 6611.1484375 disk.sda2.in‚0) disk./.usagePercent‚39.82342530584913& disk.nvme0n1.out‚13341.07861328125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 159.50390625 disk.sda.in‚0.109375' proc.memory.availableMB‚ 147.1171875* proc.memory.percent‚16.331791877746582 disk.rbd1.out‚ 11.51953125# disk.dm-0.out‚13341.07861328125 disk.drbd2165.out‚0 cpu‚26.845263744848648" disk.nvme0c0n1.in‚ 6611.1484375 disk.rbd0.out‚0 disk.sda.out‚ 159.50390625$ disk./.usageGB‚593.3978157043457 disk.dm-1.in‚0.109375 proc.cpu.threads‚17 network.sent‚ 1.3941377e+07 network.recv‚ 1.498158e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 159.50390625 disk.sda3.in‚0.109375£áËè‚(:Ý §½íÒç†æŒ& disk.nvme0n1.out‚13343.88623046875 disk.sda1.out‚0 disk.rbd1.out‚ 11.51953125 disk.nvme0n1.in‚ 6611.1484375 disk.sda1.in‚0 proc.cpu.threads‚17$ memory_percent‚99.10211563110352 disk.sda3.out‚ 159.69140625 disk.sda.in‚0.109375 network.sent‚ 1.4003934e+07$ disk./.usageGB‚593.3968162536621 disk.dm-0.in‚ 6611.1484375 disk.dm-1.in‚0.109375 disk.sda.out‚ 159.69140625 network.recv‚ 1.502557e+06* disk./.usagePercent‚39.823358231875154 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚13343.88623046875* proc.memory.percent‚16.331791877746582 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚13343.88623046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 159.69140625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 6611.1484375& proc.memory.availableMB‚ 147.109375# proc.memory.rssMB‚ 2675.80078125 cpu‚26.843476629892077 disk.sda2.in‚0 disk.sda3.in‚0.109375Úé(‚(:… §½íÒÈß“ª gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.695' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.739 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÝ©ô2‘- _wandbruntime‚4847  _runtime‚4847É äÈ’Â _step‚836 & lr-AdamW/pg1‚0.0007301394445942857 & lr-AdamW/pg2‚0.0007301394445942857 trainer/global_step‚10349 % _timestamp‚1.7843729064309926e+09 _runtime‚4847.874876438Q!Í“Ç trainer/global_step‚10349 % _timestamp‚1.7843729064309926e+09 _runtime‚4847.874876438 _step‚836 & lr-AdamW/pg1‚0.0007301394445942857 & lr-AdamW/pg2‚0.0007301394445942857ÄÚD+2”- _wandbruntime‚4847  _runtime‚4847ˆ—ž·Õ'•ÏO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378623875676160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378623674468352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47665805 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚401736952043520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚395638938750976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98161237 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395057730879115 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6076527116173 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 _runtime‚4847.879581918 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6300005984256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _step‚837 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71116166 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175726849 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171214 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171214 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚401736847067136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6097996253184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚395638955790336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401736847067136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚395638955790336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6097908316160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77555633 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6299829680128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77555635 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47665808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024–YÔÚ'–ÔO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6097996253184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 _runtime‚4847.879581918 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6300005984256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _step‚837 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71116166 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175726849 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171214 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171214 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚401736847067136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6097996253184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚395638955790336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401736847067136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚395638955790336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6097908316160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77555633 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6299829680128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77555635 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47665808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378623875676160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378623674468352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47665805 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚401736952043520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚395638938750976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98161237 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395057730879115 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6076527116173 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401774632374272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77562856 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98180509 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170518 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23452275 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378659178606592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389017778974720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47670289 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77562854 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6300594406400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175743365 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378659375612928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚389017761935360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚838 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170518 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175733372 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77562856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395094873412535 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47670292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77562854 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6098565619712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71122564 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395094770235973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175743365 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚395676066754560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401774527390720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚395676049715200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚395676066754560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6077008300613 trainer/global_step‚10349 % _timestamp‚1.7843729067695518e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6098477675520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71122801æ˜_ä&™ÞM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47670292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77562854 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6098565619712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71122564 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395094770235973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175743365 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚395676066754560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401774527390720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚395676049715200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚395676066754560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6077008300613 trainer/global_step‚10349 % _timestamp‚1.7843729067695518e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6098477675520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71122801 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372358756302848 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 _runtime‚4848.213710769 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401774632374272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98180509 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23452509 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6077094437815 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401774632374272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77562856 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98180509 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170518 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23452275 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378659178606592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389017778974720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47670289 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77562854 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6300594406400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175743365 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378659375612928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚389017761935360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚838 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170518 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175733372 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77562856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395094873412535 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143Æœª÷ó2š- _wandbruntime‚4848  _runtime‚4848_I< ›† 0loss/FRt1/assignment_loss‚1.5991128327193147 1loss/SRqqt1/assignment_loss‚0.595367086020631 trainer/global_step‚10349 _step‚839 4loss/FBt1/assignment_loss‚0.00048083948896836167 /loss/FRt1/detection_loss‚0.5178746183653282 1loss/SRqqt2/detection_loss‚0.6367910614626223 /loss/FBt1/detection_loss‚0.4827219476302465 $ _timestamp‚1.784372906771265e+09 0loss/FRt2/assignment_loss‚0.7358524844977274 2loss/FBt2/assignment_loss‚0.021712151923066025 /loss/FRt2/detection_loss‚1.6108399018752233 1loss/SRqqt1/detection_loss‚0.8297782793661447 3loss/SRqqt2/assignment_loss‚0.49673878579134123 /loss/FBt2/detection_loss‚0.4348142430913248 %loss/total_loss‚7.962084232231939 epoch‚12 _runtime‚4848.214808049Žtœ‹ 3loss/SRqqt2/assignment_loss‚0.49673878579134123 /loss/FBt2/detection_loss‚0.4348142430913248 %loss/total_loss‚7.962084232231939 epoch‚12 _runtime‚4848.214808049 0loss/FRt1/assignment_loss‚1.5991128327193147 1loss/SRqqt1/assignment_loss‚0.595367086020631 trainer/global_step‚10349 _step‚839 4loss/FBt1/assignment_loss‚0.00048083948896836167 /loss/FRt1/detection_loss‚0.5178746183653282 1loss/SRqqt2/detection_loss‚0.6367910614626223 /loss/FBt1/detection_loss‚0.4827219476302465 $ _timestamp‚1.784372906771265e+09 0loss/FRt2/assignment_loss‚0.7358524844977274 2loss/FBt2/assignment_loss‚0.021712151923066025 /loss/FRt2/detection_loss‚1.6108399018752233 1loss/SRqqt1/detection_loss‚0.8297782793661447ÇÖÄ—‘‚(:† ¶½íÒÖª®Š gpu.0.gpu‚66.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03 <¼åž‚(:Ú ¶½íÒÞòèŒ disk.rbd1.in‚0' proc.memory.availableMB‚ 147.2734375 cpu‚26.841757142653616 disk.dm-1.out‚ 160.66796875 disk.dm-0.in‚ 6611.1484375 disk.sda3.out‚ 160.66796875 disk.drbd2165.in‚0 network.sent‚ 1.400838e+07" disk.nvme0c0n1.in‚ 6611.1484375% disk.nvme0n1.out‚13350.9052734375 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 1.505774e+06* disk./.usagePercent‚39.823158802005985 disk.sda2.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2675.80078125$ memory_percent‚99.10111427307129 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0.109375" disk.dm-0.out‚13350.9052734375 disk.sda.out‚ 160.66796875' disk.nvme0c0n1.out‚13350.9052734375 disk.sda1.in‚0 disk.sda.in‚0.109375 disk.rbd1.out‚ 11.58203125* proc.memory.percent‚16.331791877746582$ disk./.usageGB‚593.3938446044922 disk.nvme0n1.in‚ 6611.1484375 disk.sda3.in‚0.109375/[Îè2Ÿ- _wandbruntime‚4869  _runtime‚4869ÛW…Ç Á _runtime‚4869.509288718 _step‚840 & lr-AdamW/pg1‚0.0007289626659495452 & lr-AdamW/pg2‚0.0007289626659495452 trainer/global_step‚10399 $ _timestamp‚1.784372928065554e+09·ÀÜæÌ¡Æ & lr-AdamW/pg1‚0.0007289626659495452 & lr-AdamW/pg2‚0.0007289626659495452 trainer/global_step‚10399 $ _timestamp‚1.784372928065554e+09 _runtime‚4869.509288718 _step‚840ÈkÍ3ä2¢- _wandbruntime‚4869  _runtime‚48694Ó&ð6&£ÎO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚211 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71448914 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380398657022464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6329236503040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397494504001536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6104925095881 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77916683 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397494486962176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23559117 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6329408612864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176543822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98637116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176553801 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390804891666432 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403620895503360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403621000479744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23558909 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98627139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47890005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6126496478208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374069420519424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397494486962176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _runtime‚4869.510622875 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77916683 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71449125 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390804908705792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380398854035968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403620895503360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47890008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6126496478208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6126408541184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396909833801673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77916685 Lò­¸ž?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98627139 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6104925095881 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77916683 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397494486962176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23559117 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6329408612864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176543822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98637116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176553801 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390804891666432 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403620895503360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403621000479744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23558909 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98627139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47890005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6126496478208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374069420519424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397494486962176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _runtime‚4869.510622875 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77916683 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71449125 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390804908705792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380398854035968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403620895503360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47890008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6126496478208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6126408541184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396909833801673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77916685 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98627139 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397531597926400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71455576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98646411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47894489 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6105406280321 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176560324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176570317 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403658680810496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71455800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397531597926400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 trainer/global_step‚10399 _step‚842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6329998292480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390841363701760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77923904 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6126977900544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390841380741120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77923906 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77923904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6126977900544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380434355230208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23561087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396946873158643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47894492 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6127065844736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23561308 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98646411 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176560324 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98636420 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6105492417523 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403658680810496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4869.852751004 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176570317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374104332034048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77923906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚224 $`6ä&§ÞM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176560324 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98636420 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6105492417523 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403658680810496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4869.852751004 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176570317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374104332034048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77923906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397531614965760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98636420 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397531597926400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71455576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98646411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47894489 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6105406280321 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176560324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176570317 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403658680810496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71455800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397531597926400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 trainer/global_step‚10399 _step‚842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6329998292480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390841363701760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77923904 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6126977900544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390841380741120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77923906 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77923904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6126977900544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380434355230208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23561087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396946873158643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47894492 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6127065844736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23561308 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98646411 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3Êø€2¨- _wandbruntime‚4869  _runtime‚4869ê5 ©‡ .loss/FRt1/detection_loss‚0.531859824100076 1loss/SRqqt1/detection_loss‚0.7421004520550084 0loss/FBt2/detection_loss‚0.46622146957034366 _runtime‚4869.853585539 _step‚843 0loss/FRt2/assignment_loss‚0.7314045936503308 epoch‚12 2loss/SRqqt1/assignment_loss‚0.5412893478238812 2loss/SRqqt2/assignment_loss‚0.5374596901564462 4loss/FBt1/assignment_loss‚4.9050874929987584e-05 1loss/SRqqt2/detection_loss‚0.5828415026612458 /loss/FBt1/detection_loss‚0.6953638941049576 0loss/FRt1/assignment_loss‚1.5905505459542195 2loss/FBt2/assignment_loss‚0.005348179220581502 /loss/FRt2/detection_loss‚1.5499436328029061 trainer/global_step‚10399 % _timestamp‚1.7843729284099483e+09 %loss/total_loss‚7.974432182974927l÷&ªŒ 0loss/FRt1/assignment_loss‚1.5905505459542195 2loss/FBt2/assignment_loss‚0.005348179220581502 /loss/FRt2/detection_loss‚1.5499436328029061 trainer/global_step‚10399 % _timestamp‚1.7843729284099483e+09 %loss/total_loss‚7.974432182974927 .loss/FRt1/detection_loss‚0.531859824100076 1loss/SRqqt1/detection_loss‚0.7421004520550084 0loss/FBt2/detection_loss‚0.46622146957034366 _runtime‚4869.853585539 _step‚843 0loss/FRt2/assignment_loss‚0.7314045936503308 epoch‚12 2loss/SRqqt1/assignment_loss‚0.5412893478238812 2loss/SRqqt2/assignment_loss‚0.5374596901564462 4loss/FBt1/assignment_loss‚4.9050874929987584e-05 1loss/SRqqt2/detection_loss‚0.5828415026612458 /loss/FBt1/detection_loss‚0.6953638941049576Ë[SЗ«‚(:… ŽíÒàËŽŒ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚162.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.519 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹‚‰5m¬‚(:Ø ŽíÒÿ°¿Ž disk.sda3.out‚ 162.359375& disk.nvme0n1.out‚13358.06787109375 disk.rbd1.Ÿ”kvout‚ 11.58203125# disk.nvme0c0n1.in‚ 6617.41015625 disk.rbd0.in‚0 network.sent‚ 1.4071788e+07 disk.rbd1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2675.80078125 disk.sda3.in‚0.109375( disk.nvme0c0n1.out‚13358.06787109375 disk.rbd0.out‚0 disk.dm-1.out‚ 162.359375! disk.nvme0n1.in‚ 6617.41015625 disk.dm-1.in‚0.109375 proc.cpu.threads‚17$ disk./.usageGB‚593.3928451538086 disk.sda1.in‚0& proc.memory.availableMB‚ 147.234375 network.recv‚ 1.510507e+06 disk.sda.in‚0.109375 disk.sda.out‚ 162.359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0# disk.dm-0.out‚13358.06787109375$ memory_percent‚99.10135269165039 disk.sda2.out‚0 disk.dm-0.in‚ 6617.41015625* proc.memory.percent‚16.331791877746582 cpu‚26.83987662587646) disk./.usagePercent‚39.82309172803201üF£‘­‚(:† Ô½íÒÁò¯Š gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚197.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.4866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×E@€ß®‚(:Ô Ô½íÒÇ£‰& disk.nvme0c0n1.out‚13365.728515625 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 163.2421875 disk.sda1.in‚0 network.recv‚ 1.514842e+06$ disk.nvme0n1.out‚13365.728515625 disk.rbd0.out‚0 disk.sda.out‚ 163.2421875! disk.nvme0n1.in‚ 6686.13671875 disk.sda2.in‚0! disk.dm-0.out‚13365.728515625 proc.memory.rssMB‚ 2689.28125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 6686.13671875 disk.rbd0.in‚0 disk.sda3.in‚0.109375 disk.sda.in‚0.109375$ memory_percent‚99.18067455291748 network.sent‚ 1.4133833e+07* disk./.usagePercent‚39.822982668822426$ disk./.usageGB‚593.3912200927734 disk.sda2.out‚0 disk.dm-0.in‚ 6686.13671875 disk.sda3.out‚ 163.2421875 cpu‚26.84993845472328 disk.rbd1.out‚ 11.64453125 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 134.23828125) proc.memory.percent‚16.41407012939453 disk.rbd1.in‚0Й2¯- _wandbruntime‚4894  _runtime‚48949åÐ+¬=°¦{ 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6026991605758667 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831498265266418 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 +jet/accuracy_2_of_2‚0.15198583900928497 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006539349909871817 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 *loss/val_total_loss‚7.8867044546483305 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8071304559707642 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18379062414169312 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6187414526939392 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6215152740478516 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8938595652580261 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7422971725463867 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 jet/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9757275581359863 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0970064103603363 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9090909361839294 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚4894.530664499 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_1‚0.3013586401939392 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8764355778694153 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6536506414413452 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6604645848274231 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16858525574207306 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6540528535842896 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6280239224433899 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6662834286689758 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.917387843132019 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1743171215057373 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21171115338802338 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.903157114982605 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09307237714529037 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9550153017044067 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9089438319206238 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1527230441570282 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (particle/accuracy‚0.8506518006324768 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9956830143928528 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899723529815674 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 $ _timestamp‚1.784372953085948e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19336600601673126 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7032310962677002 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5260416865348816 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16910196840763092 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28937000036239624 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6656302213668823 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19684307277202606 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1606924831867218 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48634639382362366 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_1_of_2‚0.773758590221405 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16183903813362122 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12923668324947357 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7799999713897705 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897671937942505 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04474730044603348 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6972343325614929 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925132989883423 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981350898742676 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.950617253780365 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01616513356566429 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7668853998184204 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963958263397217 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08021627366542816 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4284366965293884 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6549975275993347 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4431818127632141 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952272772789001 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7605013251304626 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18304938077926636 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970803260803223 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7301928400993347 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16134707629680634 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8934599757194519 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 &particle/f_score‚0.646403431892395 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8938595652580261 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878638386726379 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5788336992263794 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7034799456596375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18269237875938416 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16858525574207306 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 *jet/accuracy_1_of_4‚0.9701985716819763 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8166666030883789 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985022246837616 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_5‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9090909361839294 _step‚844 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1687752604484558 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +particle/specificity‚0.9094389081001282 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7079026699066162 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24862472712993622 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3325551152229309 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09809643775224686 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.730561375617981 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011236479505896568 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9186091423034668 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_3‚0.5567522048950195 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3707306385040283 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1686580777168274 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6552063822746277 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48965680599212646 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940239191055298 particle/accuracy_2_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09237168729305267 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.950617253780365 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7614909410476685 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4404199719429016 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16030335426330566 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1950821876525879 particle/accuracy_3_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚10412 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6504356861114502 *jet/accuracy_1_of_3‚0.9202527403831482 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1751924306154251 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984505772590637 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6402882933616638 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7672450542449951 +jet/accuracy_3_of_4‚0.09535973519086838 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8965811133384705 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6288004517555237 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 5 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19159035384655 *jet/accuracy_2_of_4‚0.7724364995956421 ,jet/accuracy_4_of_4‚0.016514042392373085 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3472222089767456 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 +particle/sensitivity‚0.6357231140136719 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9582291841506958 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3654402196407318 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15468749403953552 6validation_average_jet_accuracy‚0.3502237796783447 jet/accuracy_4_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23837502300739288 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968348741531372 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6314114928245544 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6083971858024597 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7267031669616699 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.031918201595544815 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6408957242965698 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43827158212661743 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6208766102790833 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.513527512550354 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08021627366542816 particle/accuracy_1_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6955904960632324 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5686100721359253 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6674754023551941 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14586058259010315 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.08105812221765518 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028753388673067093 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7289209365844727 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6883969306945801 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6208766102790833 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1649477183818817 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6111111044883728 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚12 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.654297411441803 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4982636868953705 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14220735430717468 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7036980986595154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1828513741493225 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8071304559707642ÊÞ·¶9±«{ 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.950617253780365 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01616513356566429 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7668853998184204 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963958263397217 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08021627366542816 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4284366965293884 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6549975275993347 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4431818127632141 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952272772789001 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7605013251304626 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18304938077926636 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970803260803223 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7301928400993347 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16134707629680634 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8934599757194519 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 &particle/f_score‚0.646403431892395 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8938595652580261 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878638386726379 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5788336992263794 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7034799456596375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18269237875938416 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16858525574207306 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 *jet/accuracy_1_of_4‚0.9701985716819763 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8166666030883789 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985022246837616 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_5‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9090909361839294 _step‚844 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1687752604484558 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +particle/specificity‚0.9094389081001282 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7079026699066162 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24862472712993622 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3325551152229309 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09809643775224686 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.730561375617981 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011236479505896568 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9186091423034668 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_3‚0.5567522048950195 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3707306385040283 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1686580777168274 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6552063822746277 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48965680599212646 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940239191055298 particle/accuracy_2_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09237168729305267 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.950617253780365 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7614909410476685 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4404199719429016 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16030335426330566 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1950821876525879 particle/accuracy_3_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚10412 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6504356861114502 *jet/accuracy_1_of_3‚0.9202527403831482 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1751924306154251 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984505772590637 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6402882933616638 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7672450542449951 +jet/accuracy_3_of_4‚0.09535973519086838 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8965811133384705 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6288004517555237 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 5 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19159035384655 *jet/accuracy_2_of_4‚0.7724364995956421 ,jet/accuracy_4_of_4‚0.016514042392373085 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3472222089767456 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 +particle/sensitivity‚0.6357231140136719 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9582291841506958 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3654402196407318 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15468749403953552 6validation_average_jet_accuracy‚0.3502237796783447 jet/accuracy_4_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23837502300739288 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968348741531372 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6314114928245544 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6083971858024597 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7267031669616699 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.031918201595544815 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6408957242965698 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43827158212661743 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6208766102790833 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.513527512550354 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08021627366542816 particle/accuracy_1_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6955904960632324 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5686100721359253 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6674754023551941 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14586058259010315 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.08105812221765518 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028753388673067093 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7289209365844727 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6883969306945801 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6208766102790833 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1649477183818817 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6111111044883728 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚12 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.654297411441803 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4982636868953705 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14220735430717468 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7036980986595154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1828513741493225 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8071304559707642 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6026991605758667 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831498265266418 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 +jet/accuracy_2_of_2‚0.15198583900928497 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006539349909871817 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 *loss/val_total_loss‚7.8867044546483305 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8071304559707642 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18379062414169312 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6187414526939392 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6215152740478516 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8938595652580261 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7422971725463867 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 jet/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9757275581359863 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0970064103603363 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9090909361839294 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚4894.530664499 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_1‚0.3013586401939392 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8764355778694153 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6536506414413452 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6604645848274231 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16858525574207306 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6540528535842896 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6280239224433899 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6662834286689758 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.917387843132019 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1743171215057373 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21171115338802338 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.903157114982605 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09307237714529037 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9550153017044067 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9089438319206238 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1527230441570282 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (particle/accuracy‚0.8506518006324768 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9956830143928528 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899723529815674 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 $ _timestamp‚1.784372953085948e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19336600601673126 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7032310962677002 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5260416865348816 8 Purity/2FRt0Sßì…ŸRqqt0FBt/FRt_purity‚0.16910196840763092 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28937000036239624 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6656302213668823 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19684307277202606 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1606924831867218 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48634639382362366 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_1_of_2‚0.773758590221405 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16183903813362122 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12923668324947357 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7799999713897705 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897671937942505 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04474730044603348 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6972343325614929 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925132989883423 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981350898742676 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129̨Õ=v²‚(:… ã½íÒͯŠ gpu.0.gpu‚66.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚185.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.117 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cZœ•è³‚(:Ý ã½íÒèÁ¢Œ disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6686.13671875 disk.sda1.out‚0 disk.sda.in‚0.109375 disk.sda.out‚ 163.50390625# memory_percent‚98.8100528717041 disk.rbd0.out‚0 disk.dm-0.in‚ 6686.13671875# proc.memory.rssMB‚ 2693.05078125 network.recv‚ 1.518555e+06 disk.rbd1.out‚ 11.64453125( disk.nvme0c0n1.out‚13499.50439453125& disk.nvme0n1.out‚13499.50439453125 disk.sda1.in‚0 disk.dm-1.out‚ 163.50390625! disk.nvme0n1.in‚ 6686.13671875 disk.sda3.out‚ 163.50390625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚26.851519102805653 network.sent‚ 1.413943e+07 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375( disk./.usagePercent‚39.8232763094642 disk.sda2.out‚0# disk.dm-0.out‚13499.50439453125' proc.memory.availableMB‚ 194.9609375* proc.memory.percent‚16.437077522277832$ disk./.usageGB‚593.3955955505371 proc.cpu.threads‚17¿òJ2´- _wandbruntime‚4911  _runtime‚4911€÷ëȵ & lr-AdamW/pg1‚0.0007277810031949741 & lr-AdamW/pg2‚0.0007277810031949741 trainer/global_step‚10449 % _timestamp‚1.7843729701520243e+09 _runtime‚4911.595857246 _step‚845i34©Í¶Ç & lr-AdamW/pg2‚0.0007277810031949741 trainer/global_step‚10449 % _timestamp‚1.7843729701520243e+09 _runtime‚4911.595857246 _step‚845 & lr-AdamW/pg1‚0.0007277810031949741ÍbÈk2·- _wandbruntime‚4911  _runtime‚4911$³‡Ô'¸ÎO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177748880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6171422623744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843729701530664e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚383104706184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71954821 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406482437150720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚399735540023147 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400311031566336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400311031566336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78512261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400311014526976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78512263 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48218785 _runtime‚ 4911.60057582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23735811 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99246596 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406482542127104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6171422623744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177758859 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99236619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48218788 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚383104909489152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6149811677035 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99236619 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6171510560768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6375374910464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393585711306752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376729509675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 trainer/global_step‚10449 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406482437150720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99246596 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71954596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23736033 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚406482542127104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192:-¡‹Ù'¹ÓO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843729701530664e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚383104706184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71954821 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406482437150720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚399735540023147 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400311031566336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400311031566336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78512261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400311014526976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78512263 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48218785 _runtime‚ 4911.60057582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23735811 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99246596 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406482542127104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6171422623744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177758859 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99236619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48218788 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚383104909489152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6149811677035 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99236619 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6171510560768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6375374910464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393585711306752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376729509675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 trainer/global_step‚10449 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406482437150720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99246596 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71954596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23736033 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚406482542127104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177748880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6171422623744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192Îà<'~2º- _wandbruntime‚4911  _runtime‚4911šgêÄ#»ÙM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78519482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400348142530560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6171991983104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23738173 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚10449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406520222457856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48223269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚383140203629568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _runtime‚4911.937253935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177765382 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71961207 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99245900 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177775375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400348125491200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6150378998677 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400348125491200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406520222457856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376764421189632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393622183342080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406520117474304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48223272 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177775375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99245900 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6375782439936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚399772476203555 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71961445 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99255891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843729704931526e+09 ?5DeviceStatsMonitor.on_tÞìrain_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78519482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6172079927296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146ZË›ä&¼ÞM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400348142530560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6171991983104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23738173 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 trainer/global_step‚10449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406520222457856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48223269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚383140203629568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 _runtime‚4911.937253935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177765382 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71961207 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99245900 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177775375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400348125491200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6150378998677 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400348125491200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406520222457856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376764421189632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393622183342080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406520117474304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48223272 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177775375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99245900 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6375782439936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚399772476203555 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71961445 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99255891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843729704931526e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78519482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6172079927296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78519482ÏýMd2½- _wandbruntime‚4911  _runtime‚4911´‡== ¾† % _timestamp‚1.7843729704950159e+09 _step‚848 0loss/FRt2/assignment_loss‚0.7796466554996472 2loss/FBt2/assignment_loss‚0.055423840870467915 epoch‚13 2loss/SRqqt1/assignment_loss‚0.6104258584945456 3loss/SRqqt2/assignment_loss‚0.45833291722259584 1loss/SRqqt2/detection_loss‚0.5806558049356297 /loss/FBt1/detection_loss‚0.5785808704536537 trainer/global_step‚10449 1loss/SRqqt1/detection_loss‚0.8206106911733514 _runtime‚4911.938628038 /loss/FBt2/detection_loss‚0.4873900172780409 %loss/total_loss‚8.024862727979194 /loss/FRt1/assignment_loss‚1.617791983181845 3loss/FBt1/assignment_loss‚0.0005220336019026354 /loss/FRt1/detection_loss‚0.5243924664956029 /loss/FRt2/detection_loss‚1.5110895887719116«‘‹Ë¿‹ /loss/FBt1/detection_loss‚0.5785808704536537 trainer/global_step‚10449 1loss/SRqqt1/detection_loss‚0.8206106911733514 _runtime‚4911.938628038 /loss/FBt2/detection_loss‚0.4873900172780409 %loss/total_loss‚8.024862727979194 /loss/FRt1/assignment_loss‚1.617791983181845 3loss/FBt1/assignment_loss‚0.0005220336019026354 /loss/FRt1/detection_loss‚0.5243924664956029 /loss/FRt2/detection_loss‚1.5110895887719116 % _timestamp‚1.7843729704950159e+09 _step‚848 0loss/FRt2/assignment_loss‚0.7796466554996472 2loss/FBt2/assignment_loss‚0.055423840870467915 epoch‚13 2loss/SRqqt1/assignment_loss‚0.6104258584945456 3loss/SRqqt2/assignment_loss‚0.45833291722259584 1loss/SRqqt2/detection_loss‚0.5806558049356297Ð-{¼øœÀ‚(:‘ ò½íÒù‹ÂŠ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o‚;åÁ‚(:Ú ò½íÒ„Û»Ž# disk.nvme0c0n1.in‚ 6686.13671875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.109375& disk.nvme0c0n1.out‚13508.458984375 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 163.83203125$ memory_percent‚98.80995750427246# proc.memory.rssMB‚ 2693.05078125) disk./.usagePercent‚39.82330498230804! disk.nvme0n1.in‚ 6686.13671875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0.109375 disk.sda1.in‚0* proc.memory.percent‚16.437077522277832 disk.drbd2165.in‚0$ disk.nvme0n1.out‚13508.458984375! disk.dm-0.out‚13508.458984375' proc.memory.availableMB‚ 194.9765625 disk.sda3.out‚ 163.83203125 disk.sda.out‚ 163.83203125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 6686.13671875 cpu‚26.849771387568737 disk.rbd1.in‚0 disk.rbd1.out‚ 11.70703125 disk.sda2.out‚0 network.sent‚ 1.4196873e+07 network.recv‚ 1.522954e+06$ disk./.usageGB‚593.3960227966309PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚408366485586944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚408366590563328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402166579777536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402166579777536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78873311 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72288291 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401587436592805 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚850 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186527744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378475110309888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161624064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178575832 trainer/global_step‚10499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384879932287488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402166562738176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99712498 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6200010785792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72288064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚408366485586944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6199922848768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78873313 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48442988 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378475085406208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6404660353536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843729921058726e+09 _runtime‚4933.549869554 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401587539769255 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99702521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6200010785792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99702521 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6404821977600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6178209656743 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99712498 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395409330112512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚408366590563328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6178123519653ìÞ÷s ÇÔO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚408366590563328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402166579777536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402166579777536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78873311 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72288291 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401587436592805 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚850 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186527744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378475110309888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161624064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178575832 trainer/global_step‚10499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384879932287488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402166562738176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99712498 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6200010785792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72288064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚408366485586944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6199922848768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78873313 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48442988 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378475085406208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6404660353536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 % _timestamp‚1.7843729921058726e+09 _runtime‚4933.549869554 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401587539769255 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99702521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6200010785792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99702521 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6404821977600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6178209656743 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99712498 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395409330112512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚408366590563328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6178123519653 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚408366485586944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146Ò) 42È- _wandbruntime‚4933  _runtime‚4933]µF›ß&ÉÙM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78880534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402203690741760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚402203690741760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72294911 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78880532 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48447472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 _step‚851 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178602327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6200492208128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99711802 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402203673702400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401624579126225 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78880532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚408404165910528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23847439 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395445802147840 trainer/global_step‚10499 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48447469 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384915440805376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72294690 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6405418980864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23847221 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378510021824512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99711802 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6178776978385 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 % _timestamp‚1.7843729924468782e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99721793 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395445785108480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6178690841183 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178602327 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78880534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402203690741760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚402203690741760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72294911 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78880532 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48447472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 _step‚851 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178602327 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6200492208128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99711802 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402203673702400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401624579126225 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78880532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚408404165910528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23847439 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395445802147840 trainer/global_step‚10499 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48447469 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384915440805376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72294690 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6405418980864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23847221 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378510021824512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99711802 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6178776978385 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 % _timestamp‚1.7843729924468782e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99721793 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395445785108480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6178690841183 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178602327 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100168423 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410250534023168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100178400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚403439436338913 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚10549 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404022110949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6206607636451 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179402784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 _runtime‚4955.316540941 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100178400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _step‚854 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48667185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79234363 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380220686041088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6206521499361 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410250638999552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410250638999552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6434123598336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404022127988736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 60816701448Ø.ÇÞ'×ØO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380220661137408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6434293611008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79234361 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397232914839552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397232931878912 % _timestamp‚1.7843730138725564e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚410250534023168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6228511010816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410250534023168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100178400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚403439436338913 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 trainer/global_step‚10549 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404022110949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6206607636451 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179402784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 _runtime‚4955.316540941 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100178400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _step‚854 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48667185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79234363 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380220686041088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6206521499361 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410250638999552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410250638999552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚220 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6434123598336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404022127988736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23953760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386654979652096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100168423 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72627490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 _step‚855 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6207174958093 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚410288214346752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48671672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 % _timestamp‚1.7843730142109945e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179429279 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397269403914240 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚10549 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48671669 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6434884720128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79241582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179419286 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79241582 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6207088820891 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100187695 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100177704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404059221913600 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179419286 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6228992433152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6229080377344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397269386874880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72627718 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6228992433152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404059238952960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386690283172352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _runtime‚4955.655051984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101mè¦ÚâM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404059238952960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23956046 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79241584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚403476578872333 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 trainer/global_step‚10549 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48671669 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6434884720128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79241582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179419286 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79241582 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6207088820891 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100187695 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100177704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404059221913600 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179419286 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6228992433152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6229080377344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397269386874880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72627718 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6228992433152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404059238952960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386690283172352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _runtime‚4955.655051984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6229080377344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72627490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 _step‚855 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6207174958093 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚410288214346752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48671672 OGDeviceStatsMonitor.on_train_batch_end/i÷ IBnactive_split.small_pool.current‚225 % _timestamp‚1.7843730142109945e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179429279 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397269403914240 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚10599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24063056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405291436085021 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381966261772288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6234919479069 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412134582459392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6463764697088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186527744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405291539261471 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48891385 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180239715 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381966236868608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79595413 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412134687435776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161624064ÈdãÞ'åØO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100634325 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412134687435776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6463603073024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100644302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100644302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6256923298816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180239715 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381966236868608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79595413 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412134687435776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161624064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405877676199936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6256923298816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚10599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24063056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405291436085021 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381966261772288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6234919479069 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412134582459392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6463764697088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186527744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405291539261471 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48891385 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79602632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382001173286912 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180256231 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412172367766528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100653597 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79602634 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405328475441879 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚4977.611897714 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6257580602368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6257492658176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388465339009024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6464190625792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405328578618441 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180246238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6235572937801 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XJDeviceStatsMonitor.on_trØñ²+ ain_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72960930 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412172367766528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24065061 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399092988641280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚405914787164160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6257580602368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388465533918208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405914770124800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382001148383232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180256231 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100653597 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6235486800599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880A¬Ösè&èâM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180246238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6235572937801 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72960930 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412172367766528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24065061 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399092988641280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚405914787164160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6257580602368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388465533918208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405914770124800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382001148383232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180256231 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100653597 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6235486800599 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79602632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382001173286912 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180256231 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412172367766528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100653597 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79602634 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405328475441879 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚4977.611897714 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6257580602368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6257492658176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388465339009024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6464190625792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405328578618441Û~vJn2é- _wandbruntime‚4977  _runtime‚4977Ã%Èp ê„ _step‚860 3loss/FBt2/assignment_loss‚0.0049464031581922885 1loss/SRqqt1/detection_loss‚0.8298559405090701 2loss/SRqqt1/assignment_loss‚0.6520328917740901 1loss/FBt1/assignment_loss‚0.01991838954212266 trainer/global_step‚10599 _runtime‚ 4977.61297674 0loss/FRt2/assignment_loss‚0.5984102457761764 2loss/SRqqt2/assignment_loss‚0.4301384265468097 1loss/SRqqt2/detection_loss‚0.6682033732909114 /loss/FBt1/detection_loss‚0.8048404445871711 % _timestamp‚1.7843730361694489e+09 0loss/FRt1/assignment_loss‚1.5311668702336245 /loss/FRt1/detection_loss‚0.6083510182273504 /loss/FRt2/detection_loss‚1.5101089836936443 /loss/FBt2/detection_loss‚0.5016003666076295 %loss/total_loss‚8.159573353946794 epoch‚13ß;­6ë‰ % _timestamp‚1.7843730361694489e+09 0loss/FRt1/assignment_loss‚1.5311668702336245 /loss/FRt1/detection_loss‚0.6083510182273504 /loss/FRt2/detection_loss‚1.5101089836936443 /loss/FBt2/detection_loss‚0.5016003666076295 %loss/total_loss‚8.159573353946794 epoch‚13 _step‚860 3loss/FBt2/assignment_loss‚0.0049464031581922885 1loss/SRqqt1/detection_loss‚0.8298559405090701 2loss/SRqqt1/assignment_loss‚0.6520328917740901 1loss/FBt1/assignment_loss‚0.01991838954212266 trainer/global_step‚10599 _runtime‚ 4977.61297674 0loss/FRt2/assignment_loss‚0.5984102457761764 2loss/SRqqt2/assignment_loss‚0.4301384265468097 1loss/SRqqt2/detection_loss‚0.6682033732909114 /loss/FBt1/detection_loss‚0.8048404445871711Ü¥ávVœì‚(:‘ ®¾íÒø§ÀŠ gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚188.512' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.702400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Påçí‚(:Ü ®¾íÒ˜ÀÑŒ$ memory_percent‚99.20883178710938 cpu‚26.84330464108052* disk./.usagePercent‚39.823485211612144 disk.dm-1.out‚ 164.87109375 disk.dm-0.in‚ 6748.78515625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.437077522277832 disk.rbd1.out‚ 11.83203125! disk.nvme0n1.in‚ 6748.78515625# proc.memory.availableMB‚129.625# proc.memory.rssMB‚ 2693.05078125 disk.sda1.out‚0 disk.sda3.out‚ 164.87109375 disk.sda.out‚ 164.87109375& disk.nvme0n1.out‚13526.05615234375 disk.sda.in‚0.109375( disk.nvme0c0n1.out‚13526.05615234375 disk.sda2.in‚0# disk.dm-0.out‚13526.05615234375# disk.nvme0c0n1.in‚ 6748.78515625 disk.rbd0.out‚0$ disk./.usageGB‚593.3987083435059 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.539922e+06 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.rbd1.in‚0 network.sent‚ 1.4390444e+07 disk.sda3.in‚0.109375ô|1Ù‘î‚(:† ½¾íÒߘŠ gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚186.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ŸC=Üï‚(:Ñ ½¾íÒà‹±Œ disk.dm-0.in‚ 6748.78515625 disk.sda3.out‚ 165.140625# proc.memory.rssMB‚ 2693.05078125 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6748.78515625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 165.140625 disk.drbd2165.out‚0) disk./.usagePercent‚39.82347881142378 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚26.845019813340652% disk.nvme0c0n1.out‚13530.59765625 disk.sda2.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚13530.59765625$ memory_percent‚99.20904636383057 network.sent‚ 1.4453167e+07 network.recv‚ 1.54457e+06$ disk./.usageGB‚593.3986129760742! disk.nvme0n1.in‚ 6748.78515625 disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 disk.dm-1.out‚ 165.140625 disk.rbd1.out‚ 11.83203125# disk.nvme0n1.out‚13530.59765625 disk.rbd0.in‚0 disk.sda.in‚0.109375( proc.memory.availableMB‚ 129.58984375* proc.memory.percent‚16.437077522277832P¯Fk2ð- _wandbruntime‚4999  _runtime‚4999k74íÇñÁ _runtime‚4999.668617454 _step‚861 & lr-AdamW/pg1‚0.0007230058907799925 & lr-AdamW/pg2‚0.0007230058907799925 trainer/global_step‚10649 $ _timestamp‚1.784373058224681e+09á 0ÛÌòÆ trainer/global_step‚10649 $ _timestamp‚1.784373058224681e+09 _runtime‚4999.668617454 _step‚861 & lr-AdamW/pg1‚0.0007230058907799925 & lr-AdamW/pg2‚0.0007230058907799925Ý$Øß!2ó- _wandbruntime‚4999  _runtime‚4999åÂŒÒÙ'ôÓO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414018630895616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414018630895616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79956463 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79956461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101110204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181056688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414018735872000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390204876630528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6493229849088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407143539007579 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24171823 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400880118372352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49115588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚407143435831129 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚10649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6285423523840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383711812599808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263403595867 _step‚862 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79956461 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407733207371776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6285511460864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4999.669864308 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73287411 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79956463 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 % _timestamp‚1.7843730582258594e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414018735872000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360i8yõØO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49115588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚407143435831129 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 trainer/global_step‚10649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6285423523840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383711812599808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263403595867 _step‚862 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79956461 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407733207371776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6285511460864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4999.669864308 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73287411 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79956463 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 % _timestamp‚1.7843730582258594e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414018735872000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414018630895616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414018630895616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79956463 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 H>DeviceSt™à5eatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79956461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101110204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181056688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414018735872000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390204876630528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6493229849088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407143539007579 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24171823 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400880118372352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090Þ·Ñ‹–2ö- _wandbruntime‚5000  _runtime‚5000¦—[ã&÷ÝM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414056416202752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 trainer/global_step‚10649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79963682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73294076 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6286080827392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79963682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚383746749018112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6493822891520 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚383746724114432 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414056416202752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6286080827392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181083183 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73293841 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414056311219200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚407770318336000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181083183 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407770318336000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49120072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407770335375360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263884780307 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843730585694413e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79963684 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181073190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24173772 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400916590407680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400916607447040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101119499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407770335375360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101119499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49120069 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24174004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6493654983168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚863 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181073190 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6263970917509ÙN¾Éè&øâM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6285992883200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73293841 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414056311219200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚407770318336000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181083183 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407770318336000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49120072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407770335375360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263884780307 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843730585694413e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79963684 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181073190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24173772 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400916590407680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400916607447040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101119499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407770335375360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101119499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49120069 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24174004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6493654983168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚863 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181073190 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6263970917509 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414056416202752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 trainer/global_step‚10649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79963682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73294076 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6286080827392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79963682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚383746749018112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6493822891520 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚383746724114432 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414056416202752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6286080827392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181083183 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415902679331840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80317513 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391980069181952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚408995435577237 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 % _timestamp‚1.7843730807738454e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181893619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385457388331008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6313923748864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73620903 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0iZþ›Þ'ƒØO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391980069181952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚408995435577237 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 % _timestamp‚1.7843730807738454e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181893619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385457388331008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6313923748864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73620903 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6314011685888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6291801575575 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101576106 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6313923748864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5022.217854273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49339785 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391979863779840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415902679331840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80317513 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80324732 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181910135 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409625866547200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49344272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409032578110657 _step‚867 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73627520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409625883586560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409625866547200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101585401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6292368897217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181910135 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73627760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181900142 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6314581052416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101575410 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80324734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24283251 trainer/global_step‚10699 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415940359655424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402740192174080 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843730811162574e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392015359054336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415940464638976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385492324749312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392015568643584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409625883586560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181900142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192æ-¬™†âM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80324732 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181910135 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409625866547200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49344272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409032578110657 _step‚867 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73627520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409625883586560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409625866547200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101585401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6292368897217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181910135 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73627760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181900142 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6314581052416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batùƒOch_end/active.small_pool.freed‚ 101575410 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80324734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24283251 trainer/global_step‚10699 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415940359655424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402740192174080 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843730811162574e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392015359054336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415940464638976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385492324749312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392015568643584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409625883586560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181900142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6314493108224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6523243894272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880ãn¦îÐ2‡- _wandbruntime‚5022  _runtime‚5022`P ˆ… 2loss/SRqqt2/assignment_loss‚0.4841939883369927 /loss/FRt2/detection_loss‚1.4705388660583159 0loss/SRqqt1/detection_loss‚0.784466481539718 1loss/SRqqt2/detection_loss‚0.6349096809604795 trainer/global_step‚10699 % _timestamp‚1.7843730811179905e+09 0loss/FRt1/assignment_loss‚1.5095602258430538 2loss/SRqqt1/assignment_loss‚0.5177306289935703 2loss/FBt2/assignment_loss‚7.07290043645881e-05 /loss/FRt1/detection_loss‚0.5358942169778995 /loss/FBt1/detection_loss‚0.5742131978930796 %loss/total_loss‚7.702853148081954 epoch‚13 _step‚868 _runtime‚5022.561529577 0loss/FRt2/assignment_loss‚0.8253973641833245 2loss/FBt1/assignment_loss‚0.002339537757466517 0loss/FBt2/detection_loss‚0.36353823053368906=Œ‘#‰Š 0loss/FRt2/assignment_loss‚0.8253973641833245 2loss/FBt1/assignment_loss‚0.002339537757466517 0loss/FBt2/detection_loss‚0.36353823053368906 2loss/SRqqt2/assignment_loss‚0.4841939883369927 /loss/FRt2/detection_loss‚1.4705388660583159 0loss/SRqqt1/detection_loss‚0.784466481539718 1loss/SRqqt2/detection_loss‚0.6349096809604795 trainer/global_step‚10699 % _timestamp‚1.7843730811179905e+09 0loss/FRt1/assignment_loss‚1.5095602258430538 2loss/SRqqt1/assignment_loss‚0.5177306289935703 2loss/FBt2/assignment_loss‚7.07290043645881e-05 /loss/FRt1/detection_loss‚0.5358942169778995 /loss/FBt1/detection_loss‚0.5742131978930796 %loss/total_loss‚7.702853148081954 epoch‚13 _step‚868 _runtime‚5022.561529577äÊ«âÊ‘Š‚(:† Û¾íÒÀ“¹Š gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚189.042' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F†ì:Þ‹‚(:Ó Û¾íÒŒº¹ disk.sda.out‚ 165.74609375 disk.nvme0n1.in‚ 6813.6875 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 6813.6875 disk.sda2.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚13541.75341796875 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.dm-1.in‚0.109375 cpu‚26.848303304798478# disk./.usageGB‚593.398609161377 disk.dm-0.in‚ 6813.6875' proc.memory.availableMB‚ 129.5859375$ memory_percent‚99.20907020568848 network.sent‚ 1.4521009e+07 network.recv‚ 1.552383e+06 disk.sda3.out‚ 165.74609375 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚13541.75341796875# proc.memory.rssMB‚ 2693.05078125) disk./.usagePercent‚39.82347855541625 disk.sda3.in‚0.109375 disk.rbd1.in‚0 disk.rbd1.out‚ 11.89453125& disk.nvme0n1.out‚13541.75341796875 disk.sda1.in‚0 disk.dm-1.out‚ 165.74609375* proc.memory.percent‚16.437077522277832¾Ô&‘Œ‚(:† ê¾íÒ–Ö‚Š gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚189.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YSGÓ‚(:È ê¾íÒÃÔƒŒ disk.dm-0.in‚ 6813.6875# disk.dm-0.out‚13548.73486328125 network.recv‚ 1.556156e+06 disk.sda3.in‚0.109375 disk.sda3.out‚ 165.90625 disk.sda1.out‚0 network.sent‚ 1.4583552e+07) disk./.usagePercent‚39.82347804340118 disk.drbd2165.out‚0& disk.nvme0n1.out‚13548.73486328125 disk.dm-1.in‚0.109375 cpu‚26.84985320376973$ disk./.usageGB‚593.3986015319824 disk.sda.out‚ 165.90625 disk.rbd1.out‚ 11.95703125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 6813.6875# memory_percent‚99.2091178894043 disk.nvme0c0n1.in‚ 6813.6875 disk.sda.in‚0.109375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 165.90625& proc.memory.availableMB‚ 129.578125( disk.nvme0c0n1.out‚13548.73486328125 disk.rbd1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2693.05078125* proc.memory.percent‚16.437077522277832 disk.sda2.in‚0Ã~v32Ž- _wandbruntime‚5044  _runtime‚5044çE-È & lr-AdamW/pg2‚0.0007205895256955679 trainer/global_step‚10749 % _timestamp‚1.7843731031686578e+09 _runtime‚5044.612543986 _step‚869 & lr-AdamW/pg1‚0.0007205895256955679+öÉÍÇ & lr-AdamW/pg1‚0.0007205895256955679 & lr-AdamW/pg2‚0.0007205895256955679 trainer/global_step‚10749 % _timestamp‚1.7843731031686578e+09 _runtime‚5044.612543986 _step‚869å[D2‘- _wandbruntime‚5044  _runtime‚5044Ìc¢Ù'’ÓO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102032031 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 _step‚870 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 _runtime‚5044.613802982 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6342423973888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102042008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417786727768064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393755086195712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚387202964062208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6342511910912 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411444320833536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182720571 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843731031697748e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24389740 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80678561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80678561 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417786832744448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6320199555283 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49563988 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚410847538499795 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80678563 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411444320833536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417786832744448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411444303794176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚387202988965888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6342511910912 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411444320833536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182720571 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 % _timestamp‚1.7843731031697748e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24389740 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80678561 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80678561 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417786832744448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6320199555283 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49563988 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚410847538499795 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80678563 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411444320833536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417786832744448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411444303794176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚387202988965888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417786727768064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393755086195712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚387202964062208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192æÇ—ïÏ2”- _wandbruntime‚5044  _runtime‚5044#Ô ? •ÝM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102041312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417824513075200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6343081277440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6320680739723 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49568469 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393790592020480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚387237875576832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411481414758400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182727094 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80685782 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24392129 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚871 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5044.955530473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6342993333248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80685782 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182727094 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552691539968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80685784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182737087 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73960368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24391899 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417824408091648 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417824513075200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411481431797760¹úa¸è&–âM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80685782 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182727094 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552691539968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80685784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182737087 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73960368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24391899 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417824408091648 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417824513075200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411481431797760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102041312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417824513075200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6343081277440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6320680739723 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49568469 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393790592020480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚387237875576832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411481414758400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182727094 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80685782 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24392129 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚871 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5044.955530473 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6342993333248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ç³àv…2—- _wandbruntime‚5044  _runtime‚5044tß8t ˜„ 2loss/SRqqt1/assignment_loss‚0.5633470594668368 2loss/SRqqt2/assignment_loss‚0.5063799126354592 4loss/FBt1/assignment_loss‚1.0958327521848332e-05 .loss/FBt1/detection_loss‚0.472188657947949 0loss/FBt2/detection_loss‚0.36408165424183325 1loss/SRqqt2/detection_loss‚0.6102341902602849 trainer/global_step‚10749 %loss/total_loss‚7.353784700120258 0loss/FRt2/assignment_loss‚0.5445178794734021 1loss/FBt2/assignment_loss‚0.04180168252907264 1loss/SRqqt1/detection_loss‚0.8267924285294622 _runtime‚5044.956640187 _step‚872 0loss/FRt1/assignment_loss‚1.5092530970449571 .loss/FRt1/detection_loss‚0.520511941988791 /loss/FRt2/detection_loss‚1.3946652376746878 $ _timestamp‚1.784373103513084e+09 epoch‚13q0‘f™‰ 0loss/FBt2/detection_loss‚0.36408165424183325 1loss/SRqqt2/detection_loss‚0.6102341902602849 trainer/global_step‚10749 %loss/total_loss‚7.353784700120258 0loss/FRt2/assignment_loss‚0.5445178794734021 1loss/FBt2/assignment_loss‚0.04180168252907264 1loss/SRqqt1/detection_loss‚0.8267924285294622 _runtime‚5044.956640187 _step‚872 0loss/FRt1/assignment_loss‚1.5092530970449571 .loss/FRt1/detection_loss‚0.520511941988791 /loss/FRt2/detection_loss‚1.3946652376746878 $ _timestamp‚1.784373103513084e+09 epoch‚13 2loss/SRqqt1/assignment_loss‚0.5633470594668368 2loss/SRqqt2/assignment_loss‚0.5063799126354592 4loss/FBt1/assignment_loss‚1.0958327521848332e-05 .loss/FBt1/detection_loss‚0.472188657947949èÑmˆ„œš‚(:‘ ù¾íÒ¸¼ãŠ gpu.0.gpu‚61.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Je·Ý›‚(:Ò ù¾íÒü çŒ proc.cpu.threads‚17 disk.dm-0.in‚ 6813.6875 disk.rbd1.out‚ 11.95703125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.458851e+07 disk.sda2.out‚0 disk.dm-1.out‚ 166.12890625# disk.dm-0.out‚13550.96044921875 disk.sda3.in‚0.109375 disk.sda3.out‚ 166.12890625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚13550.96044921875 disk.nvme0n1.in‚ 6813.6875& disk.nvme0n1.out‚13550.96044921875 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.437077522277832 cpu‚26.85151734846671$ disk./.usageGB‚593.3988876342773 disk.sda2.in‚0 disk.sda.in‚0.109375$ memory_percent‚99.20792579650879 network.recv‚ 1.559587e+06 disk.rbd1.in‚0) disk./.usagePercent‚39.82349724396625 disk.dm-1.in‚0.109375 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 6813.6875# proc.memory.rssMB‚ 2693.05078125 disk.sda.out‚ 166.12890625' proc.memory.availableMB‚ 129.7734375V…ZŽ2œ- _wandbruntime‚5067  _runtime‚5067RåÔ}ÆÀ & lr-AdamW/pg1‚0.0007193742084630189 & lr-AdamW/pg2‚0.0007193742084630189 trainer/global_step‚10799 $ _timestamp‚1.784373125757654e+09 _runtime‚ 5067.20154735 _step‚873$µµ¥ËžÅ _runtime‚ 5067.20154735 _step‚873 & lr-AdamW/pg1‚0.0007193742084630189 & lr-AdamW/pg2‚0.0007193742084630189 trainer/global_step‚10799 $ _timestamp‚1.784373125757654e+09é"òÃÄ2Ÿ- _wandbruntime‚5067  _runtime‚5067#rfÙ' ÓO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183537544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406350940710912 % _timestamp‚1.7843731257588024e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81039611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81039613 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388948564697088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419670881180672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413299869044736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24498510 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 _runtime‚5067.202888762 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _step‚874 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419670776204288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413299869044736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6371012135936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183547523 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6581582877184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6370924198912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74286460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6581404475904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102507910 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388948539793408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412699538245903 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413299852005376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395530147574272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6370924198912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6371012135936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395529944269312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102497933 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚412699435069453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406350940710912 % _timestamp‚1.7843731257588024e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81039611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81039613 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388948564697088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419670881180672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413299869044736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24498510 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 _runtime‚5067.202888762 L=DeviceStatsMonitor.on_train_b÷fÒ5Watch_start/active_bytes.all.peak‚ 6081670144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 _step‚874 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419670776204288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413299869044736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6371012135936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183547523 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6581582877184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6370924198912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74286460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6581404475904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102507910 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388948539793408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412699538245903 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413299852005376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395530147574272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6370924198912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6371012135936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395529944269312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102497933 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚412699435069453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JÈ2¢- _wandbruntime‚5067  _runtime‚5067–÷‹äâ&£ÜM B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6371581502464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419708456527872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102507214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81046834 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚10799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419708561511424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49792672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81046834 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102517205 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49792669 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6349164856633 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406387412746240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6371493558272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413336980008960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6371493558272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74293375 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413336962969600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102507214 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419708561511424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6581992577536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74293159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6582166777344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183564039 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412736474426311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24500703 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6371581502464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183564039 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388983451308032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102517205 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388983476211712 $ _timestamp‚1.784373126103979e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412736577602873 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81046832 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413336962969600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183554046 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81046832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395565443885568…‹½ýç&¤áM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6371493558272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413336980008960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6371493558272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74293375 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413336962969600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102507214 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419708561511424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6581992577536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74293159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6582166777344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183564039 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412736474426311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24500703 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6371581502464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183564039 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388983451308032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102517205 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388983476211712 $ _timestamp‚1.784373126103979e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412736577602873 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81046832 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413336962969600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183554046 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81046832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395565443885568 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6371581502464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419708456527872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102507214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81046834 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚10799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419708561511424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49792672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚875 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81046834 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102517205 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49792669 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6349164856633 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406387412746240ë¥TÒ2¥- _wandbruntime‚5067  _runtime‚5067v-è§ ¦ƒ 0loss/FRt2/assignment_loss‚0.6434909451652218 2loss/SRqqt2/assignment_loss‚0.4987261483743678 2loss/FBt1/assignment_loss‚0.018582630215314092 /loss/FBt1/detection_loss‚0.7369948706279198 .loss/FBt2/detection_loss‚0.550480534359979 $loss/total_loss‚8.32123948916203 % _timestamp‚1.7843731261055596e+09 2loss/SRqqt1/assignment_loss‚0.6584283575377966 /loss/FRt1/detection_loss‚0.5829074952183138 /loss/FRt2/detection_loss‚1.4991429347444225 1loss/SRqqt1/detection_loss‚0.7849073702171561 epoch‚13 _step‚876 trainer/global_step‚10799 0loss/FRt1/assignment_loss‚1.6635876460298027 2loss/FBt2/assignment_loss‚0.010908784932315665 1loss/SRqqt2/detection_loss‚0.6730817717394199 _runtime‚5067.549096913y„4‡§ˆ /loss/FBt1/detection_loss‚0.7369948706279198 .loss/FBt2/detection_loss‚0.550480534359979 $loss/total_loss‚8.32123948916203 % _timestamp‚1.7843731261055596e+09 2loss/SRqqt1/assignment_loss‚0.6584283575377966 /loss/FRt1/detection_loss‚0.5829074952183138 /loss/FRt2/detection_loss‚1.4991429347444225 1loss/SRqqt1/detection_loss‚0.7849073702171561 epoch‚13 _step‚876 trainer/global_step‚10799 0loss/FRt1/assignment_loss‚1.6635876460298027 2loss/FBt2/assignment_loss‚0.010908784932315665 1loss/SRqqt2/detection_loss‚0.6730817717394199 _runtime‚5067.549096913 0loss/FRt2/assignment_loss‚0.6434909451652218 2loss/SRqqt2/assignment_loss‚0.4987261483743678 2loss/FBt1/assignment_loss‚0.018582630215314092ìÅ[ò(œ¨‚(:‘ ˆ¿íÒô’Š gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚188.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.706599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ã°pØ©‚(:Í ˆ¿íÒó‹âŒ disk.sda2.out‚0 disk.nvme0n1.in‚ 6854.8359375) disk./.usagePercent‚39.82348879571762 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 6854.8359375 disk.sda3.out‚166.625 proc.cpu.threads‚17 network.recv‚ 1.564152e+06 disk.sda2.in‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0& disk.nvme0n1.out‚13554.81494140625& proc.memory.availableMB‚ 129.453125# proc.memory.rssMB‚ 2693.05078125 disk.rbd1.in‚0( disk.nvme0c0n1.out‚13554.81494140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚26.853278940759232 disk.sda.out‚166.625$ memory_percent‚99.20988082885742$ disk./.usageGB‚593.3987617492676 disk.sda3.in‚0.109375 disk.rbd1.out‚ 12.01953125* proc.memory.percent‚16.437077522277832 disk.dm-1.out‚166.625 disk.drbd2165.out‚0 network.sent‚ 1.4651572e+07# disk.dm-0.out‚13554.81494140625" disk.nvme0c0n1.in‚ 6854.8359375 disk.sda1.out‚0ê{‘ª‚(:† —¿íÒÙǨŠ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨^ æ«‚(:Û —¿íÒ¤™¬Œ( proc.memory.availableMB‚ 129.42578125$ disk./.usageGB‚593.3987770080566 disk.dm-0.in‚ 6854.8359375# disk.dm-0.out‚13558.03271484375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.01953125 network.recv‚ 1.568671e+06* disk./.usagePercent‚39.823489819747756 disk.sda3.out‚ 166.9765625 disk.drbd2165.in‚0& disk.nvme0n1.out‚13558.03271484375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2693.05078125* proc.memory.percent‚16.437077522277832 network.sent‚ 1.4714297e+07 proc.cpu.threads‚17 disk.sda.in‚0.109375" disk.nvme0c0n1.in‚ 6854.8359375 disk.sda1.out‚0 disk.dm-1.out‚ 166.9765625 disk.sda.out‚ 166.9765625( disk.nvme0c0n1.out‚13558.03271484375 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6854.8359375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375 cpu‚26.854936483050526 disk.sda2.in‚0 disk.dm-1.in‚0.109375# memory_percent‚99.2100477218628ΠçO2¬- _wandbruntime‚5089  _runtime‚50899`fÈ­Â & lr-AdamW/pg1‚0.0007181541606992482 & lr-AdamW/pg2‚0.0007181541606992482 trainer/global_step‚10849 % _timestamp‚1.7843731481576774e+09 _runtime‚5089.601571262 _step‚877„ ÿ=®Ç _step‚877 & lr-Adê43¯amW/pg1‚0.0007181541606992482 & lr-AdamW/pg2‚0.0007181541606992482 trainer/global_step‚10849 % _timestamp‚1.7843731481576774e+09 _runtime‚5089.601571262íº×~2¯- _wandbruntime‚5089  _runtime‚5089o0ç»Ø'°ÒO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81400661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6399424423936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6399512360960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74619566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421554824640512 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81400663 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6376909377609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6399424423936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102963835 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24606957 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6611082507776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102973812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414551537992011 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390694115524608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81400661 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102963835 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81400663 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408174542477312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421554824640512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚5089.602954032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184364496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74619342 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6399512360960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415155417255936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚878 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102973812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50012385 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6376995514699 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390694140428288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚414551434815561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421554929616896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415155400216576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 trainer/global_step‚10849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421554929616896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397305222936064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 $ _timestamp‚1.784373148158902e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚408174525437952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184374475 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50012388 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6610912495104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415155400216576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102973812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50012385 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6376995514699 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390694140428288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚414551434815561 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421554929616896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415155400216576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 trainer/global_step‚10849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421554929616896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397305222936064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 $ _timestamp‚1.784373148158902e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚408174525437952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184374475 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50012388 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6610912495104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415155400216576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421554824640512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _runtime‚5089.602954032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184364496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74619342 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6399512360960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415155417255936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚878 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384îEÿã2²- _wandbruntime‚5089  _runtime‚5089£ö¨ã&³ÝM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415192528220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390729051942912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102983107 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6399993783296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415192511180800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184380998 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74626120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24609044 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6611672052224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421592504964096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421592609947648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6400081727488 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5089.943061361 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 % _timestamp‚1.7843731484990327e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81407884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397340533280256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415192511180800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184390991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚204 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚879 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421592609947648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165811200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6377476699139 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102983107 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81407884 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415192528220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184390991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50016872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414588577348981 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚408210997473280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74625913 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚408211014512640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6399993783296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50016869 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184380998 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0à%K$´âM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6399993783296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50016869 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184380998 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415192528220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390729051942912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102983107 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6399993783296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415192511180800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184380998 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74626120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pAG7ëÄool.freed‚24609044 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6611672052224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421592504964096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421592609947648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6400081727488 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5089.943061361 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 % _timestamp‚1.7843731484990327e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81407884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397340533280256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415192511180800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184390991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚204 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚879 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421592609947648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165811200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6377476699139 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102983107 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81407884 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415192528220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184390991 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50016872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414588577348981 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚408210997473280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74625913 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚408211014512640ï¾>çù2µ- _wandbruntime‚5089  _runtime‚5089°t× ¶… %loss/total_loss‚7.726202323996955 2loss/SRqqt2/assignment_loss‚0.5824894330067871 3loss/FBt1/assignment_loss‚5.139134410082523e-05 3loss/FBt2/assignment_loss‚0.0012342026437569085 1loss/SRqqt1/detection_loss‚0.6509039422776052 % _timestamp‚1.7843731485008917e+09 0loss/FRt1/assignment_loss‚1.5176530866631532 .loss/FRt1/detection_loss‚0.524408584363113 0loss/FBt1/detection_loss‚0.43878249036656186 trainer/global_step‚10849 _runtime‚5089.944434103 _step‚880 0loss/FRt2/assignment_loss‚0.7534770445218162 epoch‚13 2loss/SRqqt1/assignment_loss‚0.6966734277164622 /loss/FRt2/detection_loss‚1.5101056066908376 0loss/SRqqt2/detection_loss‚0.777624291271677 .loss/FBt2/detection_loss‚0.272798823131084s6’³·Š epoch‚13 2loss/SRqqt1/assignment_loss‚0.6966734277164622 /loss/FRt2/detection_loss‚1.5101056066908376 0loss/SRqqt2/detection_loss‚0.777624291271677 .loss/FBt2/detection_loss‚0.272798823131084 %loss/total_loss‚7.726202323996955 2loss/SRqqt2/assignment_loss‚0.5824894330067871 3loss/FBt1/assignment_loss‚5.139134410082523e-05 3loss/FBt2/assignment_loss‚0.0012342026437569085 1loss/SRqqt1/detection_loss‚0.6509039422776052 % _timestamp‚1.7843731485008917e+09 0loss/FRt1/assignment_loss‚1.5176530866631532 .loss/FRt1/detection_loss‚0.524408584363113 0loss/FBt1/detection_loss‚0.43878249036656186 trainer/global_step‚10849 _runtime‚5089.944434103 _step‚880 0loss/FRt2/assignment_loss‚0.7534770445218162ð& ßR‘¸‚(:† ¦¿íÒß²Œ gpu.0.gpu‚60.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚181.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™<NÙ¹‚(:Î ¦¿íÒ‡«ÒŽ$ disk.nvme0c0n1.out‚ 13563.2109375" disk.nvme0c0n1.in‚ 6854.8359375 disk.sda2.out‚0# proc.memory.rssMB‚ 2693.05078125 cpu‚26.856603358064998 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.109375) disk./.usagePercent‚39.82348725967241# disk./.usageGB‚593.398738861084 disk.drbd2165.in‚0 disk.dm-1.out‚ 167.44140625' proc.memory.availableMB‚ 129.3984375 disk.sda3.in‚0.109375 disk.rbd1.out‚ 12.08203125 network.sent‚ 1.4719244e+07 network.recv‚ 1.5716e+06 disk.drbd2165.out‚0" disk.nvme0n1.out‚ 13563.2109375 disk.rbd0.out‚0 disk.dm-0.in‚ 6854.8359375 disk.sda.out‚ 167.44140625 disk.nvme0n1.in‚ 6854.8359375 disk.rbd0.in‚0 disk.dm-0.out‚ 13563.2109375$ memory_percent‚99.21021461486816 proc.cpu.threads‚17 disk.sda.in‚0.109375 disk.sda3.out‚ 167.44140625 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.437077522277832ój¬2º- _wandbruntime‚5111  _runtime‚5111ù±4ÓÇ»Á & lr-AdamW/pg1‚0.0007169294019457969 & lr-AdamW/pg2‚0.0007169294019457969 trainer/global_step‚10899 $ _timestamp‚1.784373170477015e+09 _runtime‚5111.920826597 _step‚881y]Ì¼Æ & lr-AdamW/pg2‚0.0007169294019457969 trainer/global_step‚10899 $ _timestamp‚1.784373170477015e+09 _runtime‚5111.920826597 _step‚881 & lr-AdamW/pg1‚0.0007169294019457969ñv2¶2½- _wandbruntime‚5111  _runtime‚5111Cà}Ù'¾ÓO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392439716159488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚5111.922100582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185191448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24716269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24716049 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6428012585984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74952857 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843731704781384e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6405393494407 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6428012585984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103439714 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392439691255808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6427924648960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81761711 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423438873076736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423438873076736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024O­K»Þ'¿ØO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103429737 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6640381539840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6427924648960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399080072795648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409998127204352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _step‚882 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6640557843968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417010965467136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81761713 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103439714 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423438978053120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423438873076736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423438873076736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392439716159488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 _runtime‚5111.922100582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185191448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24716269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24716049 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6428012585984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74952857 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843731704781384e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6405393494407 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6428012585984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103439714 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392439691255808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6427924648960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81761711 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417048059392000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103439018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185207950 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185207950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410034599239680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚883 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423476658383872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24718285 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399115771920896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6428494008320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185217943 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6405960816049 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103439018 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6640967949824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6428581952512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392474602770432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423476553400320 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74959354 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81768934 _runtime‚5112.262255933 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6428494008320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410034616279040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423476658383872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417048076431360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24718499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327™ðç&ÂáM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185217943 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176296960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6405960816049 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103439018 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6640967949824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6428581952512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392474602770432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423476553400320 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74959354 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81768934 _runtime‚5112.262255933 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6428494008320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410034616279040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423476658383872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417048076431360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24718499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6428581952512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81768932 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399115570720256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103449009 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6405874678847 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50241072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417048059392000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50241069 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416440577095089 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74959571 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392474627674112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185217943 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417048076431360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚10899 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417048059392000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103439018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185207950 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185207950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410034599239680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚883 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423476658383872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24718285 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399115771920896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6428494008320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144ó´¨É2Ã- _wandbruntime‚5112  _runtime‚5112ôé/³ Ä… /loss/FRt1/assignment_loss‚1.636861435655882 3loss/FBt1/assignment_loss‚0.0001315404445188051 /loss/FRt2/detection_loss‚1.4517810267177136 1loss/SRqqt1/detection_loss‚0.9198565558573708 trainer/global_step‚10899 0loss/FRt2/assignment_loss‚0.5861637577706692 1loss/FBt2/assignment_loss‚0.02342363961511905 epoch‚13 % _timestamp‚1.7843731708200972e+09 _step‚884 2loss/SRqqt1/assignment_loss‚0.6549519796916324 /loss/FRt1/detection_loss‚0.6104331273169952 0loss/FBt1/detection_loss‚0.41377755768597124 %loss/total_loss‚8.071344133650914 2loss/SRqqt2/assignment_loss‚0.5120499410574993 1loss/SRqqt2/detection_loss‚0.6648080645623763 /loss/FBt2/detection_loss‚0.5971055072751672 _runtime‚5112.263646789N"xŠÅŠ % _timestamp‚1.7843731708200972e+09 _step‚884 2loss/SRqqt1/assignment_loss‚0.6549519796916324 /loss/FRt1/detection_loss‚0.6104331273169952 0loss/FBt1/detection_loss‚0.41377755768597124 %loss/total_loss‚8.071344133650914 2loss/SRqqt2/assignment_loss‚0.5120499410574993 1loss/SRqqt2/detection_loss‚0.6648080645623763 /loss/FBt2/detection_loss‚0.5971055072751672 _runtime‚5112.263646789 /loss/FRt1/assignment_loss‚1.636861435655882 3loss/FBt1/assignment_loss‚0.0001315404445188051 /loss/FRt2/detection_loss‚1.4517810267177136 1loss/SRqqt1/detection_loss‚0.9198565558573708 trainer/global_step‚10899 0loss/FRt2/assignment_loss‚0.5861637577706692 1loss/FBt2/assignment_loss‚0.02342363961511905 epoch‚13ô‰t—”œÆ‚(:‘ µ¿íÒÑ–œŠ gpu.0.gpu‚13.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì3”‚ÞÇ‚(:Ó µ¿íÒ‘ªŒ disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 169.734375 disk.dm-0.in‚ 6854.8359375$ memory_percent‚99.20873641967773$ disk./.usageGB‚593.3924865722656# disk.dm-0.out‚13571.01220703125 disk.sda3.out‚ 169.734375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.08203125* proc.memory.percent‚16.437077522277832 disk.nvme0n1.in‚ 6854.8359375 disk.sda3.in‚0.109375 cpu‚26.85549113715018 network.sent‚ 1.478159e+07 disk.sda2.out‚0 network.recv‚ 1.576027e+06& disk.nvme0n1.out‚13571.01220703125 disk.sda2.in‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk./.usagePercent‚39.8230676633238 disk.dm-1.in‚0.109375 disk.sda.out‚ 169.734375" disk.nvme0c0n1.in‚ 6854.8359375( disk.nvme0c0n1.out‚13571.01220703125 disk.sda1.in‚0& proc.memory.availableMB‚ 129.640625# proc.memory.rssMB‚ 2693.05078125 proc.cpu.threads‚17 disk.rbd0.in‚0r@o±œÈ‚(:‘ Ä¿íÒ•º°Š gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚33.0 gpu.0.powerWatts‚176.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.376999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡E TáÉ‚(:Ö Ä¿íÒò© # disk.dm-0.out‚13585.32568359375* proc.memory.percent‚16.437077522277832$ memory_percent‚99.20933246612549 disk.sda.out‚ 171.96875 network.sent‚ 1.4844721e+07 disk.dm-1.in‚0.109375# disk.nvme0c0n1.in‚ 6919.94921875# proc.memory.rssMB‚ 2693.05078125( disk.nvme0c0n1.out‚13585.32568359375& disk.nvme0n1.out‚13585.32568359375 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 129.54296875 disk.dm-0.in‚ 6919.94921875 disk.sda3.in‚0.109375 disk.sda3.out‚ 171.96875# disk./.usageGB‚593.327808380127 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.580065e+06) disk./.usagePercent‚39.81872705558075 disk.sda2.in‚0 disk.dm-1.out‚ 171.96875 disk.rbd1.in‚0 disk.rbd1.out‚ 12.20703125! disk.nvme0n1.in‚ 6919.94921875 disk.sda1.out‚0 cpu‚26.85381708200757#ž…#2Ê- _wandbruntime‚5133  _runtime‚5133‚~(¨ÈË _step‚885 & lr-AdamW/pg1‚0.0007156999518196627 & lr-AdamW/pg2‚0.0007156999518196627 trainer/global_step‚10949 % _timestamp‚1.7843731924198864e+09 _runtime‚5133.863695203ÏÔ+ÍÌÇ & lr-AdamW/pg1‚0.0007156999518196627 & lr-AdamW/pg2‚0.0007156999518196627 trainer/global_step‚10949 % _timestamp‚1.7843731924198864e+09 _runtime‚5133.863695203 _step‚885õØ_ï92Í- _wandbruntime‚5133  _runtime‚5133 DÈïÛ!ÎÓO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6669858534400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _step‚886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418866496638976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75285982 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418255434307777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5133.864934342 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186028379 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82122761 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425322921512960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103905616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6670022255616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418255537484227 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411821746010112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418866513678336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425323026489344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82122761 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843731924209323e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6456512811008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6456512811008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6433791474115 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6433705337025 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50460785 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75286207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50460788 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186018400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418866513678336 GADev"°‚þiceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411821728970752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82122763 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394185266987008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425322921512960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103895639 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0€6ôÞ'ÏØO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚400855314146304 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6456424873984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103905616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82122763 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163721216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 trainer/global_step‚10949 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425323026489344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103895639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24825197 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394185291890688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6456424873984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24825419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400855125521408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _step‚886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418866496638976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75285982 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418255434307777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5133.864934342 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186028379 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82122761 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425322921512960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103905616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6670022255616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418255537484227 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411821746010112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418866513678336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425323026489344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82122761 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843731924209323e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6456512811008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6456512811008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6433791474115 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6433705337025 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50460785 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75286207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50460788 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186018400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418866513678336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411821728970752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82122763 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394185266987008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425322921512960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103895639 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6669858534400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186034902 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24827602 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6434272658555 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚418903607603200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6457082177536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75292648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚418292473664635 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚5134.205150227 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186044895 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6457082177536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6456994233344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82129984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186044895 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50465269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411858218045440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82129984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6670616530944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418903624642560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103914911 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394220178501632 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103914911 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400890819936256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 $ _timestamp‚1.784373192760995e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82129982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6670446525440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394220203405312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400890625027072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418292576841197 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0è¡•Üç&ÒáM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103904920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75292874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425360601836544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411858201006080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425360706820096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6456994233344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 170005504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186034902 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24827602 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6434272658555 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚418903607603200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6457082177536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75292648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚418292473664635 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚5134.205150227 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186044895 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6457082177536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6456994233344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82129984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186044895 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50465269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411858218045440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82129984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6670616530944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418903624642560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103914911 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394220178501632 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103914911 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400890819936256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 $ _timestamp‚1.784373192760995e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82129982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6670446525440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394220203405312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400890625027072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418292576841197 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194909184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420722044850176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427207074925568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6485013036032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24935138 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _step‚890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186845352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 trainer/global_step‚10999 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420107434053885 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104361541 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50684988 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186855331 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420722061889536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413645330737152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402630352329216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6484925099008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104361541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784373214085667e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82483813 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427207074925568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75620126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50684985 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6485013036032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420107537230335 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5155.529655182 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6484925099008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82483811 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6699306306048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚413645347776512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104371518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395930842718208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395930867621888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82483813 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6462103316733 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427206969949184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75619904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6462189453823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104371518 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427206969949184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82483811 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3003760640)FÝ'Ý×O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82483811 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402630149024256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚420722061889536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24934919 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420722044850176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420722044850176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427207074925568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6485013036032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24935138 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 _step‚890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186845352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 trainer/global_step‚10999 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420107434053885 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104361541 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50684988 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186855331 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420722061889536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413645330737152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402630352329216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6484925099008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104361541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784373214085667e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82483813 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427207074925568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75620126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50684985 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6485013036032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178401280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420107537230335 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5155.529655182 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6484925099008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82483811 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6699306306048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚413645347776512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104371518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395930842718208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395930867621888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82483813 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6462103316733 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427206969949184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75619904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203304960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6462189453823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104371518 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427206969949184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231úqnî&2Þ- _wandbruntime‚5155  _runtime‚5155kIñ%ßÝM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395965754232832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75626566 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186871847 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186871847 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104380813 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24937321 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6462756775465 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104370822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420144576587305 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104370822 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420759155814400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75626793 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427244755256320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402665847266816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413681802772480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420144473410743 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186520576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚891 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6485582402560 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6485494458368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚5155.868351722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420759155814400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82491032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420759172853760 % _timestamp‚1.7843732144242496e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161616896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6485582402560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186861854 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24937097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6700068130304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_poolå )<ò.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0P ;è&àâM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420144576587305 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104370822 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420759155814400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75626793 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427244755256320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402665847266816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413681802772480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420144473410743 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186520576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚891 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6485582402560 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6485494458368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 _runtime‚5155.868351722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420759155814400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82491032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420759172853760 % _timestamp‚1.7843732144242496e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161616896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6485582402560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186861854 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24937097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6700068130304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395965754232832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75626566 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186871847 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186871847 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104380813 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24937321 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6462756775465 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104370822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ûR Òï2á- _wandbruntime‚5155  _runtime‚5155C)û& ↠1loss/SRqqt1/assignment_loss‚0.571594955134711 3loss/SRqqt2/assignment_loss‚0.47571469178395304 /loss/FRt1/detection_loss‚0.5114270914838001 1loss/SRqqt1/detection_loss‚0.7593701610243634 0loss/FRt2/assignment_loss‚0.6768802426596905 /loss/FBt1/detection_loss‚0.4853352697995993 epoch‚13 _step‚892 2loss/FBt1/assignment_loss‚0.030604643071182654 /loss/FBt2/detection_loss‚0.6521931236687427 _runtime‚5155.869958365 0loss/FRt1/assignment_loss‚1.6173209819671917 4loss/FBt2/assignment_loss‚0.00043809022113438324 /loss/FRt2/detection_loss‚1.5050780918359994 1loss/SRqqt2/detection_loss‚0.6769690544894434 %loss/total_loss‚7.962926397139812 trainer/global_step‚10999 $ _timestamp‚1.784373214426422e+09 DPYã‹ /loss/FBt2/detection_loss‚0.6521931236687427 _runtime‚5155.869958365 0loss/FRt1/assignment_loss‚1.6173209819671917 4loss/FBt2/assignment_loss‚0.00043809022113438324 /loss/FRt2/detection_loss‚1.5050780918359994 1loss/SRqqt2/detection_loss‚0.6769690544894434 %loss/total_loss‚7.962926397139812 trainer/global_step‚10999 $ _timestamp‚1.784373214426422e+09 1loss/SRqqt1/assignment_loss‚0.571594955134711 3loss/SRqqt2/assignment_loss‚0.47571469178395304 /loss/FRt1/detection_loss‚0.5114270914838001 1loss/SRqqt1/detection_loss‚0.7593701610243634 0loss/FRt2/assignment_loss‚0.6768802426596905 /loss/FBt1/detection_loss‚0.4853352697995993 epoch‚13 _step‚892 2loss/FBt1/assignment_loss‚0.030604643071182654ü1¬¿œä‚(:‘ â¿íÒ⇗Œ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚191.777' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.355399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ÝÜìå‚(:á â¿íÒŠ¸ÏŽ( disk.nvme0c0n1.out‚13597.22998046875$ memory_percent‚99.20773506164551 disk.rbd1.in‚0 disk.rbd1.out‚ 12.26953125! disk.nvme0n1.in‚ 6919.94921875 disk.sda3.out‚ 172.80078125# proc.memory.rssMB‚ 2693.05078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 172.80078125 network.recv‚ 1.587292e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0.109375' proc.memory.availableMB‚ 129.8046875 disk.sda1.out‚0 disk.dm-1.in‚0.109375# disk.dm-0.out‚13597.22998046875* proc.memory.percent‚16.437077522277832 network.sent‚ 1.4911115e+07* disk./.usagePercent‚39.818700430797186$ disk./.usageGB‚593.3274116516113& disk.nvme0n1.out‚13597.22998046875 disk.sda1.in‚0 disk.dm-0.in‚ 6919.94921875 disk.sda.in‚0.109375 proc.cpu.threads‚17 disk.sda.out‚ 172.80078125# disk.nvme0c0n1.in‚ 6919.94921875 disk.sda2.out‚0 cpu‚26.850455136546778 disk.sda2.in‚0f»†ƒœæ‚(:‘ ñ¿íÒÅúŠ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.872' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.974400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—W·áç‚(:Ö ñ¿íÒúýœŒ disk.sda.out‚ 173.03125 disk.rbd1.in‚0& disk.nvme0n1.out‚13600.05908203125 disk.sda2.out‚0 disk.dm-1.in‚0.109375( disk.nvme0c0n1.out‚13600.05908203125! disk.nvme0n1.in‚ 6919.94921875( proc.memory.availableMB‚ 129.64453125 proc.cpu.threads‚17 network.sent‚ 1.49738e+07 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 173.03125 network.recv‚ 1.591556e+06 disk.sda3.out‚ 173.03125 cpu‚26.848750137292882 disk.sda2.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚39.81869710269925$ disk./.usageGB‚593.3273620605469 disk.dm-0.in‚ 6919.94921875 disk.rbd1.out‚ 12.26953125# disk.nvme0c0n1.in‚ 6919.94921875$ memory_percent‚99.20871257781982# disk.dm-0.out‚13600.05908203125 disk.sda3.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2693.05078125* proc.memory.percent‚16.437077522277832 disk.rbd0.out‚0 disk.sda1.in‚0–_32è- _wandbruntime‚5177  _runtime‚5177O+<ÕÈé _step‚893 & lr-AdamW/pg1‚0.0007132270562927282 & lr-AdamW/pg2‚0.0007132270562927282 trainer/global_step‚11049 % _timestamp‚1.7843732359071333e+09 _runtime‚5177.351056853_'ÍêÇ & lr-AdamW/pg2‚0.0007132270562927282 trainer/global_step‚11049 % _timestamp‚1.7843732359071333e+09 _runtime‚5177.351056853 _step‚893 & lr-AdamW/pg1‚0.0007132270562927282ýtáÆy2ë- _wandbruntime‚5177  _runtime‚5177–@OPÙ'ìÓO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚11049 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82844863 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75952857 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚421959536976443 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429091018385408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6513513261056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚421959433799993 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75953064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6728926527488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚422577593061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404405369880576 _runtime‚5177.355604097 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397676443353088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187682283 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6513513261056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82844861 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104827443 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104837420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚207 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6513425324032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50909188 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚415468949542912 % _timestamp‚1.7843732359083416e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429091018385408 _step‚894 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6513425324032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚422577610100736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚422577593061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82844863 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6490587433531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397676418449408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429091123361792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50909185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚204 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0æL©Õ—íØO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6513425324032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50909188 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚415468949542912 % _timestamp‚1.7843732359083416e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429091018385408 _step‚894 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6513425324032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚422577610100736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚422577593061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82844863 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6490587433531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397676418449408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429091123361792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50909185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚204 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚11049 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82844863 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75952857 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚421959536976443 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429091018385408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6513513261056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚421959433799993 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75953064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6728926527488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚422577593061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404405369880576 _runtime‚5177.355604097 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397676443353088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187682283 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6513513261056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82844861 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104827443 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104837420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚207 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0þèxm¦2î- _wandbruntime‚5177  _runtime‚5177/Gîã&ïÝM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6729344215040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚422614704025600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82852084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404440873282560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429128803692544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚422614721064960 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187698799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421996473156851 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚895 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50913669 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚429128803692544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6513994683392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25045807 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104836724 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75959691 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75959476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104836724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104846715 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82852084 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82852082 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415505404538880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6514082627584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397711329964032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6513994683392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 Eè&ðâM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187698799 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187688806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104846715 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚11049 % _timestamp‚1.7843732362477834e+09 _runtime‚5177.691812976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚422614704025600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82852084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404440873282560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429128803692544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚422614721064960 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187698799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421996473156851 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _step‚895 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50913669 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚429128803692544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6513994683392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25045807 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104836724 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75959691 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75959476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104836724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104846715 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82852084 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82852082 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415505404538880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6514082627584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397711329964032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6513994683392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 E2ö- _wandbruntime‚5198  _runtime‚5198‘*“¬È÷ & lr-AdamW/pg1‚0.0007119836505003692 & lr-AdamW/pg2‚0.0007119836505003692 trainer/global_step‚11099 % _timestamp‚1.7843732575380776e+09 _runtime‚5198.981845062 _step‚897¾¤ÜÍÍøÇ & lr-AdamW/pg2‚0.0007119836505003692 trainer/global_step‚11099 % _timestamp‚1.7843732575380776e+09 _runtime‚5198.981845062 _step‚897 & lr-AdamW/pg1‚0.0007119836505003692à)Ô…2ù- _wandbruntime‚5198  _runtime‚51987› ’úÒO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6518985413239 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25152408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6758409539072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105293345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399422019084288 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83205911 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83205913 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76285793 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6518899276149 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430975066821632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105293345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51133385 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚417292551309312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105303322 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6542013486080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚424433158311936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399421994180608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430975066821632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51133388 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423811433546101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚424433158311936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6541925549056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚423811536722551 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188509235 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83205913 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚430975171798016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_poo&  Fl.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188499256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105303322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚424433141272576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76286020 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚406180428623360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417292534269952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188499256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105303322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚424433141272576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76286020 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚406180428623360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417292534269952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430975066821632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105293345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51133385 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚417292551309312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105303322 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6542013486080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚424433158311936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399421994180608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430975066821632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51133388 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197013504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423811433546101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚424433158311936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6541925549056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚423811536722551 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188509235 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83205913 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚430975171798016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680‚¬»H2ü- _wandbruntime‚5199  _runtime‚5199¼•5mã&ýÝM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424470269276160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399456905695232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188515758 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚424470252236800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6519466597679 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83213134 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431012747145216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542494908416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6519552734881 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51137872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6542582852608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105312617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406215730636288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 % _timestamp‚1.7843732578817136e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6542582852608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25154870 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6758997043712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _runtime‚5199.325767787 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83213134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424470269276160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188525751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76292742 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51137869 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423848576079521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105302626 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83213132 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417329006305280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚899 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188525751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424470252236800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6758824941056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399456930598912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417329023344640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105302626 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431012852128768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76292502 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423848472902959 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105312617 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83213132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188515758 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431012852128768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327}¢š¥!þâM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424470269276160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399456905695232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188515758 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚424470252236800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6519466597679 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83213134 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431012747145216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542494908416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6519552734881 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51137872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6542582852608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105312617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406215730636288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 % _timestamp‚1.7843732578817136e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6542582852608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25154870 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6758997043712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 _runtime‚5199.325767787 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83213134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424470269276160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188525751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76292742 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51137869 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423848576079521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105302626 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83213132 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417329006305280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _step‚899 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188525751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424470252236800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6758824941056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399456930598912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417329023344640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105302626 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431012852128768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_en°ìç(Cd/inactive_split.all.freed‚76292502 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423848472902959 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105312617 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83213132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188515758 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431012852128768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936ƒØ̆2ÿ- _wandbruntime‚5199  _runtime‚5199|=/ €ƒ % _timestamp‚1.7843732578833444e+09 _runtime‚5199.326879556 2loss/FBt2/assignment_loss‚0.009683301487887283 trainer/global_step‚11099 _step‚900 0loss/FRt1/assignment_loss‚1.5569251202960883 0loss/FRt2/assignment_loss‚0.6871158667423758 1loss/SRqqt2/assignment_loss‚0.541474663540519 .loss/FRt2/detection_loss‚1.412727833974437 1loss/SRqqt1/detection_loss‚0.8255146192327046 1loss/SRqqt2/detection_loss‚0.6607471170391579 -loss/FBt1/detection_loss‚0.73142713388162 2loss/SRqqt1/assignment_loss‚0.5070080352028153 4loss/FBt1/assignment_loss‚0.00020625975092548017 /loss/FRt1/detection_loss‚0.6000720157203787 /loss/FBt2/detection_loss‚0.7368421692802918 %loss/total_loss‚8.269744136149201 epoch‚13Ö~vùˆ 1loss/SRqqt1/detection_loss‚0.8255146192327046 1loss/SRqqt2/detection_loss‚0.6607471170391579 -loss/FBt1/detection_loss‚0.73142713388162 2loss/SRqqt1/assignment_loss‚0.5070080352028153 4loss/FBt1/assignment_loss‚0.00020625975092548017 /loss/FRt1/detection_loss‚0.6000720157203787 /loss/FBt2/detection_loss‚0.7368421692802918 %loss/total_loss‚8.269744136149201 epoch‚13 % _timestamp‚1.7843732578833444e+09 _runtime‚5199.326879556 2loss/FBt2/assignment_loss‚0.009683301487887283 trainer/global_step‚11099 _step‚900 0loss/FRt1/assignment_loss‚1.5569251202960883 0loss/FRt2/assignment_loss‚0.6871158667423758 1loss/SRqqt2/assignment_loss‚0.541474663540519 .loss/FRt2/detection_loss‚1.412727833974437„ä–¨‘‚‚(:† ÀíÒ¨©ãŒ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚189.042' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rˆF䃂(:Ù ÀíÒÇ•²Ž disk.dm-1.out‚ 173.546875# disk.dm-0.out‚13610.85693359375 disk.sda3.out‚ 173.546875 disk.drbd2165.out‚0& disk.nvme0n1.out‚13610.85693359375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 12.33203125! disk.nvme0n1.in‚ 6996.44921875 disk.sda.out‚ 173.546875( disk.nvme0c0n1.out‚13610.85693359375' proc.memory.availableMB‚ 129.8984375 disk.sda2.in‚0* proc.memory.percent‚16.437077522277832) disk./.usagePercent‚39.81853120981706$ disk./.usageGB‚593.3248901367188 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 6996.44921875# proc.memory.rssMB‚ 2693.05078125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.rbd1.in‚0$ memory_percent‚99.20716285705566 cpu‚26.845293780316755 disk.sda3.in‚0.109375# disk.nvme0c0n1.in‚ 6996.44921875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.5042497e+07 network.recv‚ 1.59964e+06ûZ†Fœ„‚(:‘ žÀíÒö¸Š gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ƒ²ç…‚(:Ü žÀíÒ×¾´Œ! disk.nvme0n1.in‚ 6996.44921875 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚ 1.603447e+06 disk.dm-1.in‚0.109375% proc.memory.availableMB‚ 129.84375 disk.sda.in‚0.109375 disk.sda2.in‚0# proc.memory.rssMB‚ 2693.05078125 cpu‚26.84366091658852 disk.drbd2165.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6996.44921875( disk.nvme0c0n1.out‚13617.70068359375& disk.nvme0n1.out‚13617.70068359375* proc.memory.percent‚16.437077522277832 network.sent‚ 1.5104575e+07) disk./.usagePercent‚39.81858548341432 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 12.39453125 disk.sda1.in‚0 disk.dm-0.in‚ 6996.44921875$ disk./.usageGB‚593.3256988525391 disk.rbd0.out‚0 disk.dm-1.out‚ 173.80078125# memory_percent‚99.2074966430664# disk.dm-0.out‚13617.70068359375 disk.sda3.in‚0.109375 disk.sda3.out‚ 173.80078125 proc.cpu.threads‚17 disk.sda.out‚ 173.80078125ïok2†- _wandbruntime‚5220  _runtime‚5220ǺüDȇ % _timestamp‚1.7843732794412944e+09 _runtime‚5220.885068568 _step‚901 & lr-AdamW/pg1‚0.0007107356325515752 & lr-AdamW/pg2‚0.0007107356325515752 trainer/global_step‚11149Šñ6ìÍˆÇ % _timestamp‚1.7843732794412944e+09 _runtime‚5220.885068568 _step‚901 & lr-AdamW/pg1‚0.0007107356325515752 & lr-AdamW/pg2‚0.0007107356325515752 trainer/global_step‚11149… ÞÐ2‰- _wandbruntime‚5220  _runtime‚5220a†™Ø'ŠÒO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51357588 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432859220234240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105759247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425663433292209 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 trainer/global_step‚11149 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚5220.886269265 _step‚902 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105769224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401167569911808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425663536468659 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432859115257856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6570425774080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432859115257856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401167594815488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419116153075712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚426288706523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6547383392947 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6570425774080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83566963 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83566961 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25261011 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83566961 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51357585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105769224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚426288706523136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432859220234240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6787881361408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6787698765824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426288689483776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189336187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407955476176896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189326208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105759247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407955268677632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42628868948377601ðdÝ'‹×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425663433292209 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 trainer/global_step‚11149 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚5220.886269265 _step‚902 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105769224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401167569911808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425663536468659 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432859115257856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6570425774080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432859115257856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401167594815488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419116153075712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚426288706523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6547383392947 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6570425774080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83566963 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83566961 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25261011 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83566961 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51357585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105769224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚426288706523136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432859220234240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6787881361408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6787698765824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426288689483776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189336187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407955476176896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189326208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105759247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407955268677632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚426288689483776 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51357588 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432859220234240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105759247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189342710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25263121 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401202481426432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6547864577387 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843732797819138e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76625423 trainer/global_step‚11149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407990772807680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚432896795581440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚426325817487360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189352703 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419152625111040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407990965619712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6571083077632 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51362069 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25263351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚426325800448000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189342710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.f¹Ðà·reed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425700575825629 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83574182 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6571083077632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425700472649067 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚426325800448000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105768528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚432896900564992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419152608071680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76625190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51362072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6788291381248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192¤¶xè&ŽâM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189342710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425700575825629 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83574182 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6571083077632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425700472649067 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚426325800448000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105768528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚432896900564992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419152608071680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76625190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51362072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6788291381248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432896900564992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189352703 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189342710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25263121 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401202481426432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6547864577387 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7843732797819138e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76625423 trainer/global_step‚11149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407990772807680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚432896795581440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚426325817487360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189352703 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419152625111040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407990965619712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6571083077632 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51362069 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25263351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚426325800448000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101‡2&ÌW2- _wandbruntime‚5221  _runtime‚5221ŽŠCt … 1loss/SRqqt1/detection_loss‚0.8894066366528769 1loss/SRqqt2/detection_loss‚0.6640039889695439 .loss/FBt1/detection_loss‚0.571247773244977 %loss/total_loss‚8.282299458192934 0loss/FRt1/assignment_loss‚1.6160037142003891 0loss/FRt2/assignment_loss‚0.6825823371003313 2loss/SRqqt2/assignment_loss‚0.5436966322698216 3loss/FBt1/assignment_loss‚0.0010122344375369766 /loss/FRt1/detection_loss‚0.6053247850509891 trainer/global_step‚11149 2loss/SRqqt1/assignment_loss‚0.5959501828650475 1loss/FBt2/assignment_loss‚0.02221677913897213 /loss/FRt2/detection_loss‚1.6329533866538508 0loss/FBt2/detection_loss‚0.45790100760859787 _runtime‚5221.227335131 _step‚904 epoch‚13 % _timestamp‚1.7843732797838008e+09Øϳµ‘Š 3loss/FBt1/assignment_loss‚0.0010122344375369766 /loss/FRt1/detection_loss‚0.6053247850509891 trainer/global_step‚11149 2loss/SRqqt1/assignment_loss‚0.5959501828650475 1loss/FBt2/assignment_loss‚0.02221677913897213 /loss/FRt2/detection_loss‚1.6329533866538508 0loss/FBt2/detection_loss‚0.45790100760859787 _runtime‚5221.227335131 _step‚904 epoch‚13 % _timestamp‚1.7843732797838008e+09 1loss/SRqqt1/detection_loss‚0.8894066366528769 1loss/SRqqt2/detection_loss‚0.6640039889695439 .loss/FBt1/detection_loss‚0.571247773244977 %loss/total_loss‚8.282299458192934 0loss/FRt1/assignment_loss‚1.6160037142003891 0loss/FRt2/assignment_loss‚0.6825823371003313 2loss/SRqqt2/assignment_loss‚0.5436966322698216ˆ&ÿcÕœ’‚(:‘ ­ÀíÒëÙÍŠ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚162.354' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.470800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¢iÒå“‚(:Ú ­ÀíÒã•¢ disk.drbd2165.out‚0 disk.rbd1.out‚ 12.39453125! disk.nvme0n1.in‚ 6996.44921875$ disk.nvme0n1.out‚13620.775390625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 6996.44921875! disk.dm-0.out‚13620.775390625& disk.nvme0c0n1.out‚13620.775390625 disk.sda1.in‚0 disk.sda3.in‚0.109375$ memory_percent‚99.20763969421387 network.recv‚ 1.607147e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6996.44921875* proc.memory.percent‚16.437077522277832 cpu‚26.841959739958153 disk.sda3.out‚ 173.98046875 network.sent‚ 1.5110117e+07$ disk./.usageGB‚593.3255004882812 disk.dm-1.out‚ 173.98046875# proc.memory.rssMB‚ 2693.05078125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.in‚0.109375' proc.memory.availableMB‚ 129.8203125) disk./.usagePercent‚39.81857217102254 disk.rbd0.out‚0 disk.sda.out‚ 173.98046875^ʲ2”- _wandbruntime‚5242  _runtime‚5242F¤v¥È•Â % _timestamp‚1.7843733012125711e+09 _runtime‚5242.656352582 _step‚905 & lr-AdamW/pg1‚0.0007094830224358868 & lr-AdamW/pg2‚0.0007094830224358868 trainer/global_step‚11199A2ù¸Í–Ç % _timestamp‚1.7843733012125711e+09 _runtime‚5242.656352582 _step‚905 & lr-AdamW/pg1‚0.0007094830224358868 & lr-AdamW/pg2‚0.0007094830224358868 trainer/global_step‚11199‰a)Sø2—- _wandbruntime‚5242  _runtime‚5242–ÇdØ'˜ÒO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106235126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83928011 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6575695235565 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409730337169920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428144254734336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106225149 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420939754842112 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚420939737802752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6575781372655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51581788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83928013 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6598925999104 trainer/global_step‚11199 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434743163694080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434743268670464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51581785 $ _timestamp‚1.784373301213642e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6599013936128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402913145643008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83928011 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6817191526912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚434743268670464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _step‚906 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428144237694976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434743163694080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428144237694976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76952116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76951898 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6598925999104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83928013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25370328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979i !+g™×O JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6575781372655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51581788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83928013 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6598925999104 trainer/global_step‚11199 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434743163694080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434743268670464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51581785 $ _timestamp‚1.784373301213642e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6599013936128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402913145643008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83928011 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188624896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6817191526912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚434743268670464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 _step‚906 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428144237694976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434743163694080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428144237694976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76952116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76951898 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6 Õ½v598925999104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83928013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25370328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106235126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83928011 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6575695235565 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409730337169920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428144254734336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106225149 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420939754842112 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚420939737802752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Šã݆2š- _wandbruntime‚5242  _runtime‚5242Íôóúã&›ÝM _step‚907 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190179655 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428181365698560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434780949001216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106234430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6817948491264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚11199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83935234 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106244421 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190179655 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420976226877440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428181365698560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6599495358464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51586272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76958721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6599583302656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83935232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402948082061312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190169662 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434780844017664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428181348659200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76958513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51586269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25372449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434780949001216 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434780949001216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106234430 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6817948491264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚11199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83935234 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106244421 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190179655 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420976226877440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428181365698560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6599495358464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51586272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76958721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6599583302656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83935232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402948082061312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190169662 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434780844017664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428181348659200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76958513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51586269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25372449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473‹uœ2- _wandbruntime‚5242  _runtime‚5242UA' ž‡ 0loss/FRt2/assignment_loss‚0.7301882735177792 3loss/SRqqt2/assignment_loss‚0.49819968981098584 3loss/FBt1/assignment_loss‚1.860905500272736e-05 4loss/FBt2/assignment_loss‚0.00039095128267625513 /loss/FRt1/detection_loss‚0.5209368221348561 trainer/global_step‚11199 % _timestamp‚1.7843733015539045e+09 1loss/SRqqt1/detection_loss‚0.8714142733631102 %loss/total_loss‚8.140367317488685 epoch‚13 /loss/FRt2/detection_loss‚1.5774008455405313 1loss/SRqqt2/detection_loss‚0.7587483521034201 -loss/FBt2/detection_loss‚0.39229531333937 _step‚908 0loss/FRt1/assignment_loss‚1.7273600524718609 2loss/SRqqt1/assignment_loss‚0.5334889171335129 /loss/FBt1/detection_loss‚0.5299252177355811 _runtime‚5242.997446349ƒºÅ 3loss/SRqqt2/assignment_loss‚0.49819968981098584 3loss/FBt1/assignment_loss‚1.860905500272736e-05 4loss/FBt2/assignment_loss‚0.00039095128267625513 /loss/FRt1/detection_loss‚0.5209368221348561 trainer/global_step‚11199 % _timestamp‚1.7843733015539045e+09 1loss/SRqqt1/detection_loss‚0.8714142733631102 %loss/total_loss‚8.140367317488685 epoch‚13 /loss/FRt2/detection_loss‚1.5774008455405313 1loss/SRqqt2/detection_loss‚0.7587483521034201 -loss/FBt2/detection_loss‚0.39229531333937 _step‚908 0loss/FRt1/assignment_loss‚1.7273600524718609 2loss/SRqqt1/assignment_loss‚0.5334889171335129 /loss/FBt1/detection_loss‚0.5299252177355811 _runtime‚5242.997446349 0loss/FRt2/assignment_loss‚0.7301882735177792Œ>£5ˆ‘ ‚(:† ¼ÀíÒÛÜÀŠ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚170.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yrÞ¡‚(:Ó ¼ÀíÒÖ§²Ž$ memory_percent‚99.21865463256836$ disk./.usageGB‚593.3250198364258 disk.sda3.in‚0.109375 disk.rbd1.in‚0 disk.rbd1.out‚ 12.45703125 disk.sda2.in‚0! disk.dm-0.out‚13624.751953125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚13624.751953125* proc.memory.percent‚16.444087028503418 cpu‚26.840928628787253 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.dm-0.in‚ 7042.96484375 disk.rbd0.out‚0& proc.memory.availableMB‚ 128.015625 network.recv‚ 1.611415e+06 disk.sda.in‚0.109375 disk.sda.out‚ 174.453125 disk.sda3.out‚ 174.453125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7042.96484375! disk.nvme0n1.in‚ 7042.96484375# proc.memory.rssMB‚ 2694.19921875 network.sent‚ 1.5172956e+07 disk.rbd0.in‚0) disk./.usagePercent‚39.81853991407322 disk.dm-1.out‚ 174.453125$ disk.nvme0n1.out‚13624.751953125õ}yu‘¢‚(:† ËÀíÒÛ’Œ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚189.887' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.9774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÿÜ£‚(:Ñ ËÀíÒ‰áô disk.sda2.in‚0 cpu‚26.850327272442577 proc.cpu.threads‚17 disk.dm-1.in‚0.109375# disk.nvme0c0n1.in‚ 7042.96484375 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚13627.380859375! disk.nvme0n1.in‚ 7042.96484375$ disk.nvme0n1.out‚13627.380859375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.26140308380127 disk.sda3.in‚0.109375 disk.sda1.out‚0 disk.sda.in‚0.109375* proc.memory.percent‚16.489148139953613# disk./.usageGB‚593.325065612793 disk.rbd1.in‚0 disk.rbd1.out‚ 12.45703125 disk.sda1.in‚0) disk./.usagePercent‚39.81854298616363 disk.sda3.out‚ 174.84375 disk.drbd2165.in‚0( proc.memory.availableMB‚ 121.01171875# proc.memory.rssMB‚ 2701.58203125 network.sent‚ 1.5235568e+07 disk.dm-1.out‚ 174.84375 disk.sda2.out‚0 network.recv‚ 1.615769e+06 disk.dm-0.in‚ 7042.96484375! disk.dm-0.out‚13627.380859375 disk.sda.out‚ 174.84375 I¡2¤- _wandbruntime‚5268  _runtime‚5268¡ð}Ó ¥Óz 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13873256742954254 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6492588520050049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8548533916473389 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6834965348243713 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13516709208488464 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7047335505485535 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7425616383552551 /particle/accuracy_3_of_4‚0.9922618865966797 +particle/sensitivity‚0.6205869317054749 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2130461186170578 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15430976450443268 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6400688290596008 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630951046943665 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4564521908760071 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_2_of_4‚0.644788920879364 trainer/global_step‚11213 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968408942222595 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21982558071613312 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8499999046325684 $ _timestamp‚1.784373326955051e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6789707541465759 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9896848797798157 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11974241584539413 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7166603207588196 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5565104484558105 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0372212789952755 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00572529062628746 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5512198209762573 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 *jet/accuracy_2_of_3‚0.5204336047172546 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2831517159938812 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966789484024048 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4783’?Ò¶†1201813697815 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901729822158813 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48786473274230957 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11041513085365295 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030642041936516762 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38260722160339355 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6232410669326782 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7362004518508911 *particle/specificity‚0.907932460308075 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13902868330478668 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15428534150123596 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5871113538742065 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.651435375213623 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12362124025821686 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _runtime‚5268.399527366 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.016514042392373085 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.663098931312561 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11444148421287537 +jet/accuracy_3_of_4‚0.08931287378072739 +jet/accuracy_1_of_1‚0.30200907588005066 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4808900058269501 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5793696641921997 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969369530677795 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032951921224594116 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5509503483772278 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6859167218208313 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7269427180290222 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 _step‚909 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9963623881340027 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14687499403953552 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6278678178787231 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9778438806533813 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6531190872192383 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13451962172985077 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9889219403266907 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13365070521831512 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960194826126099 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.739757239818573 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14012554287910461 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1486491560935974 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47999998927116394 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6963518261909485 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6761319041252136 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6833065748214722 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.07589985430240631 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46227505803108215 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9189749360084534 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12479595839977264 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15449321269989014 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6122850179672241 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6959471702575684 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014644940383732319 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8499999046325684 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09953616559505463 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995177686214447 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9945475459098816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6570565104484558 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6904808282852173 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7425616383552551 *jet/accuracy_1_of_4‚0.9523405432701111 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7150201201438904 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023728271946310997 /particle/accuracy_4_of_4‚0.9415333271026611 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31407439708709717 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026857029646635056 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.155472531914711 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166865348816 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6583912968635559 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4135802090167999 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03312770649790764 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6231796741485596 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13915401697158813 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14012554287910461 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7300000190734863 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10683158785104752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6870051622390747 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.704281210899353 *jet/accuracy_1_of_3‚0.9040613770484924 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.880024254322052 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11730951070785522 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7321368455886841 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3858032822608948 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21578256785869598 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15203015506267548 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.85384601354599 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6570565104484558 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4922284185886383 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12382756173610687 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 'particle/accuracy‚0.846218466758728 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8684834241867065 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3454550504684448 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7495983839035034 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0936378613114357 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030954863876104355 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9455136060714722 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7178276777267456 'particle/f_score‚0.6341015696525574 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7739315032958984 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24286390841007233 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880251288414001 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7120248079299927 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957600831985474 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6679509282112122 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7064265012741089 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6074878573417664 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0372212789952755 epoch‚13 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6697785258293152 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036007899791002274 jet/accuracy_3_of_5‚1 )loss/val_total_loss‚7.851210348344562 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7300000190734863 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010386141017079353 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875Ȩ»È^=¦Øz 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6278678178787231 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9778438806533813 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6531190872192383 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13451962172985077 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9889219403266907 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13365070521831512 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960194826126099 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.739757239818573 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14012554287910461 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1486491560935974 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47999998927116394 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6963518261909485 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6761319041252136 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6833065748214722 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.07589985430240631 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46227505803108215 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9189749360084534 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12479595839977264 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15449321269989014 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6122850179672241 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6959471702575684 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014644940383732319 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8499999046325684 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09953616559505463 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995177686214447 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9945475459098816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6570565104484558 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6904808282852173 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7425616383552551 *jet/accuracy_1_of_4‚0.9523405432701111 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7150201201438904 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023728271946310997 /particle/accuracy_4_of_4‚0.9415333271026611 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31407439708709717 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026857029646635056 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.155472531914711 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166865348816 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6583912968635559 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4135802090167999 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03312770649790764 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6231796741485596 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13915401697158813 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14012554287910461 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7300000190734863 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10683158785104752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6870051622390747 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.704281210899353 *jet/accuracy_1_of_3‚0.9040613770484924 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.880024254322052 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11730951070785522 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7321368455886841 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3858032822608948 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21578256785869598 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15203015506267548 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.85384601354599 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6570565104484558 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4922284185886383 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12382756173610687 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 'particle/accuracy‚0.846218466758728 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8684834241867065 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3454550504684448 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7495983839035034 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0936378613114357 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030954863876104355 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9455136060714722 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7178276777267456 'particle/f_score‚0.6341015696525574 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7739315032958984 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24286390841007233 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880251288414001 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7120248079299927 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957600831985474 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6679509282112122 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7064265012741089 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6074878573417664 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0372212789952755 epoch‚13 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6697785258293152 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036007899791002274 jet/accuracy_3_of_5‚1 )loss/val_total_loss‚7.851210348344562 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7300000190734863 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010386141017079353 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13873256742954254 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6492588520050049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8548533916473389 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6834965348243713 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13516709208488464 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7047335505485535 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7425616383552551 /particle/accuracy_3_of_4‚0.9922618865966797 +particle/sensitivity‚0.6205869317054749 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2130461186170578 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15430976450443268 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6400688290596008 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8630951046943665 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4564521908760071 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_2_of_4‚0.644788920879364 trainer/global_step‚11213 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968408942222595 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21982558071613312 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8499999046325684 $ _timestamp‚1.784373326955051e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6789707541465759 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9896848797798157 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11974241584539413 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7166603207588196 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5565104484558105 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0372212789952755 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00572529062628746 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5512198209762573 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 *jet/accuracy_2_of_3‚0.5204336047172546 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2831517159938812 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966789484024048 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4783201813697815 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901729822158813 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48786473274230957 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11041513085365295 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030642041936516762 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38260722160339355 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6232410669326782 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7362004518508911 *particle/specificity‚0.907932460308075 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13902868330478668 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15428534150123596 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5871113538742065 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.651435375213623 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12362124025821686 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 _runtime‚5268.399527366 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.016514042392373085 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.663098931312561 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11444148421287537 +jet/accuracy_3_of_4‚0.08931287378072739 +jet/accuracy_1_of_1‚0.30200907588005066 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4808900058269501 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5793696641921997 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969369530677795 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032951921224594116 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5509503483772278 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6859167218208313 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7269427180290222 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 _step‚909 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9963623881340027 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14687499403953552 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125î´†°‘§‚(:† ÚÀíÒõò½Š gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚186.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R&œXᨂ(:Ö ÚÀíÒ…©ÒŒ network.recv‚ 1.618819e+06 disk.sda.in‚0.109375! disk.nvme0n1.in‚ 7042.96484375% disk.nvme0n1.out‚13631.8193359375 disk.dm-1.in‚0.109375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 7042.96484375" disk.dm-0.out‚13631.8193359375# disk.nvme0c0n1.in‚ 7042.96484375' disk.nvme0c0n1.out‚13631.8193359375 disk.drbd2165.in‚0 disk.rbd1.out‚ 12.58203125 disk.sda2.out‚0 disk.sda3.out‚ 175.0859375 disk.rbd0.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 117.76953125 cpu‚26.8514547829881 network.sent‚ 1.5240041e+07# disk./.usageGB‚593.325309753418 disk.sda.out‚ 175.0859375 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 175.0859375) disk./.usagePercent‚39.81855937064582 disk.sda2.in‚0# memory_percent‚99.2811918258667 disk.sda3.in‚0.109375# proc.memory.rssMB‚ 2704.20703125) proc.memory.percent‚16.50516986846924 proc.cpu.threads‚17@é72©- _wandbruntime‚5283  _runtime‚5283!sÓ ÈªÂ _step‚910 & lr-AdamW/pg1‚0.0007082258402163975 & lr-AdamW/pg2‚0.0007082258402163975 trainer/global_step‚11249 % _timestamp‚1.7843733422953794e+09 _runtime‚5283.739152307ž>éžÍ«Ç trainer/global_step‚11249 % _timestamp‚1.7843733422953794e+09 _runtime‚5283.739152307 _step‚910 & lr-AdamW/pg1‚0.0007082258402163975 & lr-AdamW/pg2‚0.0007082258402163975ŽÜBè2¬- _wandbruntime‚5283  _runtime‚5283¾R1r ­ÓO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191358213 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412436480593920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620667953804 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106844601 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412436685996032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6643940079104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430960765259776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train1½«g_batch_start/active.large_pool.allocated‚84523591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437604705338880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6863426293760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437604810315264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430960765259776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51910565 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚430341242436236 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84523591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437604705338880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6863426293760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437604810315264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430960765259776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51910565 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚430341242436236 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412436685996032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6643940079104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430960765259776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0˜\ûÝ2¯- _wandbruntime‚5284  _runtime‚5284Õ—Ûã&°ÝM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 _step‚912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405608171216896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180491264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6621149138244 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚430378178616644 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6644597382656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191374715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51915052 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84530810 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437642490646016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106843905 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191374715 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437642385662464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6644509438464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106843905 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191384708 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412472184851456 trainer/global_step‚11249 _runtime‚5284.078554737 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84530810 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84530812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77465177 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6864013634560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚430997876224000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚430378281793206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77464940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51915049 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106853896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6863833143296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843733426342258e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405608146313216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25550125 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106853896 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437642490646016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106843905 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191374715 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437642385662464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6644509438464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106843905 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191384708 \HDevi´SWeceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412472184851456 trainer/global_step‚11249 _runtime‚5284.078554737 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84530810 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84530812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77465177 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6864013634560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚430997876224000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚430378281793206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77464940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51915049 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106853896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6863833143296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843733426342258e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405608146313216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25550125 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106853896 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6672440304128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414211695411200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439488858751488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77792139 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439488753775104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚432816330510336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6892859977728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432193242182344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425544159209472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25657371 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439488753775104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6672528241152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107300526 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _step‚915 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25657147 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚5305.589829682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚432193139005894 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439488858751488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432816330510336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84884641 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77791912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚432816313470976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84884641 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52134765 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192195144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107310503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6892683673600 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192185165 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84884639 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407318835433472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6649065933512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107300526 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6672440304128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6672440304128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414211695411200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439488858751488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77792139 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439488753775104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚432816330510336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6892859977728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432193242182344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425544159209472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25657371 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439488753775104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6672528241152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107300526 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _step‚915 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25657147 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 _runtime‚5305.589829682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚432193139005894 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439488858751488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432816330510336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84884641 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77791912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚432816313470976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192“Û^$i2½- _wandbruntime‚5305  _runtime‚5305i&·F‡ ¾ÝM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192211660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚432230178362752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84891862 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚432853424435200 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107309807 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192211660 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6673009663488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84891860 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6673097607680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52139252 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77798592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192201667 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25659574 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6673009663488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84891862 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77798826 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439526434098688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84891860 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407353722044416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6673097607680 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚432853441474560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425580631244800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5305.930704028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439526539082240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192201667 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107319798 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚439526539082240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52139249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936¨F£è&¿âM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚432853441474560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425580631244800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5305.930704028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439526539082240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192201667 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107319798 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚439526539082240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52139249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192211660 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚432230178362752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84891862 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚432853424435200 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107309807 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192211660 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6673009663488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84891860 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6673097607680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52139252 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77798592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192201667 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25659574 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6673009663488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84891862 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77798826 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439526434098688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84891860 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407353722044416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6673097607680 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52358968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409064411164672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚434045138752002 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434671861682176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415986521044480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434671878721536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107776405 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52358965 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427367778015232 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6700940529152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6701028466176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6677463913220 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 trainer/global_step‚11349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107766428 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25765842 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434671878721536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6677377776130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441372802211328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193022096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441372907187712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6701028466176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434671878721536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6677377776130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441372802211328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193022096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441372907187712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6701028466176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52358968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409064411164672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚434045138752002 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434671861682176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415986521044480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434671878721536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107776405 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52358965 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427367778015232 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360—u3Ëæ2Í- _wandbruntime‚5327  _runtime‚5327‰íâ&ÎÜM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434708972646400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25767926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6922723569152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6922899866112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427404233011200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6701597832704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6701597832704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193038612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107775709 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52363449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409099322679296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚441410587518464 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚434082281285422 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427404250050560 _runtime‚5327.742055912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193028619 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85252910 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6701509888512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107785700 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85252910 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52363452 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434082178108860 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107785700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434708989685760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416022021344768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107775709 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6701509888512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78131152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143³Ð ”ç&ÏáM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85252912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434082178108860 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107785700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434708989685760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416022021344768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107775709 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6701509888512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78131152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434708972646400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25767926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6922723569152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6922899866112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427404233011200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6701597832704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6701597832704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193038612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107775709 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52363449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409099322679296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚441410587518464 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚434082281285422 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427404250050560 _runtime‚5327.742055912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193028619 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85252910 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6701509888512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107785700 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85252910 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52363452 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436527409893376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6729528691200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193849048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85606739 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443256955623936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25876023 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417761589086720 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443256955623936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85606739 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108242307 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108242307 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6951627094528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6705775755838 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85606741 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443256850647552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6729528691200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6951792912896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 trainer/global_step‚11399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6729440754176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85606741 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108232330 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417761779808768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚5349.258771218 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436527426932736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6705861892928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443256850647552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78459191 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78458963 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410809961992192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435897138498110 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436527426932736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436527409893376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25875798 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193839069 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144%ë?óÞ'ÚØO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108242307 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6951627094528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6705775755838 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85606741 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443256850647552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6729528691200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6951792912896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 trainer/global_step‚11399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6729440754176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85606741 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108232330 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417761779808768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚5349.258771218 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436527426932736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6705861892928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443256850647552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78459191 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78458963 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410809961992192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435897138498110 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436527426932736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436527409893376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25875798 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193839069 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165818368 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52583165 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190722048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108232330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52583168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6729440754176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429191379781632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436527409893376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6729528691200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193849048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85606739 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443256955623936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25876023 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417761589086720 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443256955623936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85606739 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108242307 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0›ÀÅaµ2Û- _wandbruntime‚5349  _runtime‚5349µÌ¯¶â&ÜÜM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6730098057728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435934281031530 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436564520857600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410844873506816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429227851816960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78465669 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85613960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193865564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6730098057728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6706429214570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108241611 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193865564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52587649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6952216592896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚5349.597018254 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85613962 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193855571 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784373408152989e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193855571 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436564537896960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410844898410496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6952388695552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108241611 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6730010113536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436564537896960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78465889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417797287106048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25878020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6706343077368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚11399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85613960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108251602 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6952216592896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 _runtime‚5349.597018254 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85613962 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193855571 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784373408152989e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193855571 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436564537896960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410844898410496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6952388695552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108241611 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMoni±¬^”ntor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚220 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6730010113536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436564537896960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78465889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417797287106048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25878020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6706343077368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚11399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85613960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108251602 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435934281031530 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436564520857600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410844873506816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429227851816960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78465669 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85613960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193865564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6730098057728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6706429214570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108241611 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193865564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52587649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991œ\\Êj2Þ- _wandbruntime‚5349  _runtime‚5349V†_’ ß… 0loss/FRt2/assignment_loss‚0.7824134590791508 2loss/SRqqt2/assignment_loss‚0.5024827181194246 /loss/FBt1/detection_loss‚0.4169721942479638 /loss/FBt2/detection_loss‚0.5416664563837306 % _timestamp‚1.7843734081548762e+09 _runtime‚5349.598422945 _step‚925 2loss/SRqqt1/assignment_loss‚0.6434546839150644 3loss/FBt1/assignment_loss‚0.0002608850312998098 /loss/FRt2/detection_loss‚1.6600382979262003 1loss/SRqqt2/detection_loss‚0.6341026309389118 epoch‚14 0loss/FRt1/assignment_loss‚1.5859824497267085 1loss/FBt2/assignment_loss‚0.01739027659302946 trainer/global_step‚11399 /loss/FRt1/detection_loss‚0.6500373127214498 1loss/SRqqt1/detection_loss‚0.7694147751603874 %loss/total_loss‚8.204216139843322YHÅÖàŠ trainer/global_step‚11399 /loss/FRt1/detection_loss‚0.6500373127214498 1loss/SRqqt1/detection_loss‚0.7694147751603874 %loss/total_loss‚8.204216139843322 0loss/FRt2/assignment_loss‚0.7824134590791508 2loss/SRqqt2/assignment_loss‚0.5024827181194246 /loss/FBt1/detection_loss‚0.4169721942479638 /loss/FBt2/detection_loss‚0.5416664563837306 % _timestamp‚1.7843734081548762e+09 _runtime‚5349.598422945 _step‚925 2loss/SRqqt1/assignment_loss‚0.6434546839150644 3loss/FBt1/assignment_loss‚0.0002608850312998098 /loss/FRt2/detection_loss‚1.6600382979262003 1loss/SRqqt2/detection_loss‚0.6341026309389118 epoch‚14 0loss/FRt1/assignment_loss‚1.5859824497267085 1loss/FBt2/assignment_loss‚0.01739027659302946ˆÄ£á‚(:… ¥ÁíÒ³©£Š gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚194.196' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.8392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ§ÛÃââ‚(:× ¥ÁíÒۉ͌ disk.dm-1.in‚0.109375 disk.sda3.in‚0.109375 network.recv‚ 1.639518e+06$ disk./.usageGB‚593.3238792419434 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 7086.203125# memory_percent‚99.2821455001831# proc.memory.rssMB‚ 2704.20703125 network.sent‚ 1.5490612e+07 disk.sda1.in‚0" disk.dm-0.out‚13657.6826171875 disk.sda.out‚ 176.71484375* disk./.usagePercent‚39.818463367820485 disk.sda1.out‚0 disk.dm-0.in‚ 7086.203125 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.50516986846924 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚13657.6826171875 disk.nvme0n1.in‚ 7086.203125% disk.nvme0n1.out‚13657.6826171875 disk.sda3.out‚ 176.71484375 disk.rbd1.out‚ 12.70703125 disk.sda2.out‚0 disk.dm-1.out‚ 176.71484375( proc.memory.availableMB‚ 117.61328125 cpu‚26.843121219775202 disk.rbd0.in‚0†?±‘ã‚(:† ´ÁíÒÁˆ¨Š gpu.0.gpu‚66.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘\áä‚(:Ö ´ÁíÒ©ØòŒ disk.dm-1.out‚ 177.3515625$ memory_percent‚99.28123950958252 cpu‚26.84142357656632 network.recv‚ 1.643794e+06 disk.drbd2165.in‚0 disk.rbd1.out‚ 12.76953125# disk.nvme0c0n1.in‚ 7153.76953125 disk.rbd0.in‚0 network.sent‚ 1.5553657e+07$ disk./.usageGB‚593.3237152099609 disk.sda.in‚0.109375! disk.nvme0n1.in‚ 7153.76953125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2704.20703125) disk./.usagePercent‚39.81845235949651 disk.sda3.out‚ 177.3515625$ disk.nvme0n1.out‚13663.478515625 disk.sda2.out‚0) proc.memory.percent‚16.50516986846924 disk.sda1.in‚0! disk.dm-0.out‚13663.478515625( proc.memory.availableMB‚ 117.76171875 disk.sda3.in‚0.109375 disk.sda.out‚ 177.3515625 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚13663.478515625 disk.sda2.in‚0 disk.dm-0.in‚ 7153.76953125 disk.dm-1.in‚0.109375€ V"2å- _wandbruntime‚5370  _runtime‚5370KfÈæ _runtime‚5370.911623388 _step‚926 & lr-AdamW/pg1‚0.0007031517941186223 & lr-AdamW/pg2‚0.0007031517941186223 trainer/global_step‚11449 % _timestamp‚1.7843734294677289e+09/pácÍçÇ & lr-AdamW/pg1‚0.0007031517941186223 & lr-AdamW/pg2‚0.0007031517941186223 trainer/global_step‚11449 % _timestamp‚1.7843734294677289e+09 _runtime‚5370.911623388 _step‚926žT\2è- _wandbruntime‚5370  _runtime‚5370DTª?Ù'éÓO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85967791 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85967789 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438382958104576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78792704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25985115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6734259872636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6757940979200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6981336532992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52807365 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412555562627072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚5370.916585755 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108708209 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85967791 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431014981548032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445141004060160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52807368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412555537723392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445140899083776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108708209 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _step‚927 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78792480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚438382958104576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194676000 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6758028916224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438382975143936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419536693757952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6734173735546 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194666021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚11449âEïì-'êØO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445141004060160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419536899160064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚438382975143936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445140899083776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108708209 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _step‚927 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78792480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚438382958104576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194676000 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6758028916224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438382975143936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419536693757952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6734173735546 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194666021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚11449 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85967791 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85967789 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438382958104576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78792704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25985115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6734259872636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6757940979200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6981336532992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52807365 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412555562627072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 _runtime‚5370.916585755 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108708209 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85967791 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431014981548032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445141004060160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52807368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412555537723392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438420069068800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78799369 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445178684390912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194692516 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445178579407360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52811849 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981744603136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437786280777638 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431051453583360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6734827194278 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25987517 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5371.253313358 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6758598282752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108717504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438420086108160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚11449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438420069068800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85975010 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108707513 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419572392944640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843734298092268e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚438420086108160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6734741057076 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445178684390912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6758598282752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194682523 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437786177601076 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194682523 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419572193841152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25987282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431051436544000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85975010 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6981918802944 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6758510338560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194692516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78799131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412590449238016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445178579407360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52811849 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981744603136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437786280777638 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431051453583360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6734827194278 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25987517 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5371.253313358 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6758598282752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108717504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438420086108160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚11449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438420069068800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85975010 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108707513 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419572392944640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 % _timestamp‚1.7843734298092268e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚438420086108160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6734741057076 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445178684390912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6758598282752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194682523 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437786177601076 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194682523 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419572193841152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25987282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431051436544000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85975010 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6981918802944 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6758510338560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194692516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78799131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412590449238016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438420069068800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78799369 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445178684390912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 :±HÛ2î- _wandbruntime‚5371  _runtime‚5371õÔ— ï„ trainer/global_step‚11449 _step‚929 0loss/FRt2/assignment_loss‚0.6985226237951819 /loss/FRt1/detection_loss‚0.6267874644528472 0loss/SRqqt2/detection_loss‚0.614808729582573 %loss/total_loss‚7.925137141347548 /loss/FRt1/assignment_loss‚1.560744408899405 2loss/SRqqt1/assignment_loss‚0.5418028299441818 2loss/FBt1/assignment_loss‚0.001502696648845472 1loss/FBt2/assignment_loss‚0.04920258010905456 % _timestamp‚1.7843734298109071e+09 1loss/SRqqt1/detection_loss‚0.7933298164334986 _runtime‚5371.254454021 epoch‚14 3loss/SRqqt2/assignment_loss‚0.49943655996919595 /loss/FRt2/detection_loss‚1.4847253027617635 0loss/FBt1/detection_loss‚0.43624239940196274 /loss/FBt2/detection_loss‚0.6180317293490387ú „ð‰ _runtime‚5371.254454021 epoch‚14 3loss/SRqqt2/assignment_loss‚0.49943655996919595 /loss/FRt2/detection_loss‚1.4847253027617635 0loss/FBt1/detection_loss‚0.43624239940196274 /loss/FBt2/detection_loss‚0.6180317293490387 trainer/global_step‚11449 _step‚929 0loss/FRt2/assignment_loss‚0.6985226237951819 /loss/FRt1/detection_loss‚0.6267874644528472 0loss/SRqqt2/detection_loss‚0.614808729582573 %loss/total_loss‚7.925137141347548 /loss/FRt1/assignment_loss‚1.560744408899405 2loss/SRqqt1/assignment_loss‚0.5418028299441818 2loss/FBt1/assignment_loss‚0.001502696648845472 1loss/FBt2/assignment_loss‚0.04920258010905456 % _timestamp‚1.7843734298109071e+09 1loss/SRqqt1/detection_loss‚0.7933298164334986¡Ž·õÿñ‚(:… ÃÁíÒ¦ÌÛ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÚ!JØò‚(:Í ÃÁíÒÛæ£ disk.dm-0.in‚ 7153.76953125 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 7153.76953125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.647144e+06 disk.dm-1.in‚0.109375 disk.dm-1.out‚177.6875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2704.20703125 network.sent‚ 1.5558616e+07 disk.sda2.out‚0& disk.nvme0c0n1.out‚13666.470703125 disk.rbd0.out‚0 cpu‚26.83985235444275* disk./.usagePercent‚39.818453895541715 disk.drbd2165.out‚0 disk.sda.in‚0.109375# disk.nvme0c0n1.in‚ 7153.76953125 disk.rbd0.in‚0( proc.memory.availableMB‚ 117.79296875$ memory_percent‚99.28104877471924 disk.sda2.in‚0! disk.dm-0.out‚13666.470703125 disk.sda3.in‚0.109375 disk.sda3.out‚177.6875 disk.sda.out‚177.6875 disk.rbd1.out‚ 12.7734375$ disk./.usageGB‚593.3237380981445$ disk.nvme0n1.out‚13666.470703125) proc.memory.percent‚16.50516986846924 proc.cpu.threads‚17‹¤WF2ó- _wandbruntime‚5392  _runtime‚5392<¶ºÈô & lr-AdamW/pg2‚0.0007018720548783208 trainer/global_step‚11499 % _timestamp‚1.7843734513324785e+09 _runtime‚5392.776268016 _step‚930 & lr-AdamW/pg1‚0.0007018720548783208ò¾ ÍõÇ % _timestamp‚1.7843734513324785e+09 _runtime‚5392.776268016 _step‚930 & lr-AdamW/pg1‚0.0007018720548783208 & lr-AdamW/pg2‚0.0007018720548783208 trainer/global_step‚11499¢=ü™2ö- _wandbruntime‚5392  _runtime‚5392[dëâD#÷ÓO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109164134 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26093905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26093660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6786529141248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6786441204224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86328839 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6762657852344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7010756661248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚11499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚447025052496384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439601137990326 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109174111 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440238506315776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 % _timestamp‚1.7843734513334787e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53031565 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86328841 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109164134 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79125225 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439601241166776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚432838583314432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86328839 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440238523355136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _runtime‚5392.780596197 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447025052496384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79125473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195492973 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86328841 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195502952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447024947520000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚414301138358272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414301113454592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6762571715254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109174111 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6786529141248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447024947520000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432838566275072 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473QMìÞ'øØO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26093905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26093660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6786529141248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6786441204224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172109824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86328839 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6762657852344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7010756661248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚11499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚447025052496384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439601137990326 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109174111 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440238506315776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 % _timestamp‚1.7843734513334787e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53031565 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86328841 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109164134 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79125225 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439601241166776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚432838583314432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86328839 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440238523355136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 _runtime‚5392.780596197 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447025052496384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79125473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195492973 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86328841 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195502952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447024947520000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚414301138358272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414301113454592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6762571715254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109174111 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6786529141248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447024947520000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432838566275072 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109164134 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680£"š02ù- _wandbruntime‚5393  _runtime‚5393ª½³íã&úÝM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439638177347184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440275634319360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86336062 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚432875055349760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6763225173986 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79131868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195519468 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109183406 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109173415 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚932 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109173415 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421347197446144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚414336024969216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421347396549632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447062732827136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86336060 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 trainer/global_step‚11499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195519468 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109183406 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439638280523746 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440275634319360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53036049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86336062 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚5393.117690833 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26095819 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6787010563584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440275617280000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚414336049872896 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843734516736188e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6787098507776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447062732827136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360rÑÛßè&ûâM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109183406 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109173415 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 _step‚932 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109173415 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421347197446144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚414336024969216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421347396549632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447062732827136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86336060 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 trainer/global_step‚11499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195519468 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109183406 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199103488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439638280523746 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440275634319360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53036049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86336062 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _runtime‚5393.117690833 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26095819 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6787010563584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440275617280000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚414336049872896 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 % _timestamp‚1.7843734516736188e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6787098507776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447062732827136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439638177347184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440275634319360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86336062 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚432875055349760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6763225173986 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79131868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195519468 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53255765 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434662185080832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6790969694962 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109640013 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448909100932608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441453137736434 trainer/global_step‚11549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448908995956224 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79458815 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86689891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86689889 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6814941429248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86689891 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416046714089472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6791055832052 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441453240912884 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423086734455808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442094054526976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7040221574144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448908995956224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448909100932608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6815029366272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53255768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434662168041472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196329904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26203047 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5414.820281876 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109640013 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416046689185792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 % _timestamp‚1.7843734733763216e+09 _step‚935 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448908995956224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448909100932608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6815029366272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53255768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434662168041472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196329904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26203047 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5414.820281876 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109640013 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416046689185792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 % _timestamp‚1.7843734733763216e+09 _step‚935 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53255765 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434662185080832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6790969694962 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109640013 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448909100932608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441453137736434 trainer/global_step‚11549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448908995956224 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79458815 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86689891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86689889 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6814941429248§¾mse‘‡ ‚(:† áÁíү곊 gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚168.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E¸{çˆ ‚(:Ü áÁíҹŔŽ$ memory_percent‚99.28281307220459) proc.memory.percent‚16.50516986846924 cpu‚26.838634045888092( proc.memory.availableMB‚ 117.50390625 network.recv‚ 1.655292e+06 disk.drbd2165.in‚0) disk./.usagePercent‚39.81844544729309$ disk./.usageGB‚593.3236122131348 disk.sda2.in‚0 disk.sda3.in‚0.109375# proc.memory.rssMB‚ 2704.20703125 network.sent‚ 1.5683871e+07 disk.sda1.in‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 178.30078125 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 7153.76953125' disk.nvme0c0n1.out‚13675.7314453125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0.109375 disk.dm-0.in‚ 7153.76953125! disk.nvme0n1.in‚ 7153.76953125 proc.cpu.threads‚17 disk.rbd1.out‚ 12.8359375% disk.nvme0n1.out‚13675.7314453125 disk.rbd0.out‚0 disk.sda1.out‚0" disk.dm-0.out‚13675.7314453125 disk.sda3.out‚ 178.30078125 disk.sda.out‚ 178.30078125 disk.drbd2165.out‚0ûµß2‰ - _wandbruntime‚5415  _runtime‚5415bÿ ãÙ!Š ÛM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6815510788608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196336427 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86697112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442131182530560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423122234811392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442131165491200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448946676279808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109649308 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442131182530560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6815598732800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6791537016492 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196336427 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79465179 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448946781263360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416081625604096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109639317 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53260249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26205161 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 $ _timestamp‚1.784373473722088e+09 _runtime‚ 5415.16618727 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53260252 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196346420 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196346420 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚434698657116160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚423122438109184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441490280269854 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6791623153694 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86697112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79465413 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109649308 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚11549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86697110 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448946676279808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109649308 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442131182530560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6815598732800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6791537016492 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196336427 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79465179 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448946781263360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416081625604096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109639317 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53260249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26205161 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 $ _timestamp‚1.784373473722088e+09 _runtime‚ 5415.16618727 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53260252 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196346420 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196346420 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚434698657116160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚423122438109184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441490280269854 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6791623153694 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86697112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79465413 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109649308 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 trainer/global_step‚11549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86697110 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423122234811392¨g,tB2Œ - _wandbruntime‚5415  _runtime‚5415 ¤  † 0loss/FRt2/assignment_loss‚0.6619098242556695 3loss/SRqqt2/assignment_loss‚0.45894759442609995 1loss/SRqqt1/detection_loss‚0.8437281321526224 .loss/FBt1/detection_loss‚0.926885103310148 /loss/FBt2/detection_loss‚0.6221371919692804 %loss/total_loss‚8.532761604406236 % _timestamp‚1.7843734737237184e+09 _step‚937 4loss/FBt1/assignment_loss‚0.00016886098541514608 1loss/FBt2/assignment_loss‚0.02644007462768189 /loss/FRt1/detection_loss‚0.5743919401439772 _runtime‚5415.167254154 1loss/SRqqt2/detection_loss‚0.6376339729231618 0loss/FRt1/assignment_loss‚1.6410184414715758 2loss/SRqqt1/assignment_loss‚0.5861885903834211 /loss/FRt2/detection_loss‚1.5533118777571817 epoch‚14 trainer/global_step‚11549lEêŽ ‹ epoch‚14 trainer/global_step‚11549 0loss/FRt2/assignment_loss‚0.6619098242556695 3loss/SRqqt2/assignment_loss‚0.45894759442609995 1loss/SRqqt1/detection_loss‚0.8437281321526224 .loss/FBt1/detection_loss‚0.926885103310148 /loss/FBt2/detection_loss‚0.6221371919692804 %loss/total_loss‚8.532761604406236 % _timestamp‚1.7843734737237184e+09 _step‚937 4loss/FBt1/assignment_loss‚0.00016886098541514608 1loss/FBt2/assignment_loss‚0.02644007462768189 /loss/FRt1/detection_loss‚0.5743919401439772 _runtime‚5415.167254154 1loss/SRqqt2/detection_loss‚0.6376339729231618 0loss/FRt1/assignment_loss‚1.6410184414715758 2loss/SRqqt1/assignment_loss‚0.5861885903834211 /loss/FRt2/detection_loss‚1.5533118777571817©0†Z‘ ‚(:† ðÁíҢ䱊 gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚188.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Eè ‚(:Ý ðÁíÒ±ÔóŒ disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚ 1.659311e+06 disk.dm-0.in‚ 7203.47265625 disk.rbd1.in‚0 disk.rbd1.out‚ 12.9140625 disk.rbd0.out‚0$ memory_percent‚99.28300380706787 disk.dm-1.in‚0.109375 disk.sda2.in‚0# proc.memory.rssMB‚ 2704.20703125 disk.dm-1.out‚ 178.62109375 disk.sda3.out‚ 178.62109375 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.5735802e+07* disk./.usagePercent‚39.818435719006786" disk.dm-0.out‚13681.2099609375 disk.sda3.in‚0.109375# disk.nvme0c0n1.in‚ 7203.47265625! disk.nvme0n1.in‚ 7203.47265625( proc.memory.availableMB‚ 117.47265625) proc.memory.percent‚16.50516986846924 disk.sda.in‚0.109375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚13681.2099609375 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚26.837028704684126$ disk./.usageGB‚593.3234672546387 disk.sda.out‚ 178.62109375% disk.nvme0n1.out‚13681.2099609375{ÕD2‘ - _wandbruntime‚5436  _runtime‚54367þæãÈ’  & lr-AdamW/pg1‚0.0006992992463704968 & lr-AdamW/pg2‚0.0006992992463704968 trainer/global_step‚11599 % _timestamp‚1.7843734952624674e+09 _runtime‚5436.706280647 _step‚938â)•Í“ Ç & lr-AdamW/pg2‚0.0006992992463704968 trainer/global_step‚11599 % _timestamp‚1.7843734952624674e+09 _runtime‚5436.706280647 _step‚938 & lr-AdamW/pg1‚0.0006992992463704968ª‹â~›2” - _wandbruntime‚5436  _runtime‚5436¤díOÙ'• ÓO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53479965 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443305137482542 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436485769807872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197156856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110095938 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424862002522624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚5436.707504407 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443949619777536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚417792289820672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436485786847232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450793149368832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110105915 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443305240658992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6819453811760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6843529591296 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6843441654272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443949619777536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6843441654272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87050939 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843734952635026e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79791544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26311579 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚11599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87050939 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450793044392448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424861803411968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚450793149368832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚450793044392448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6843529591296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 _step‚939 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79791779 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87050941 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7069538494976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110095938 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6819367674670 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197146877 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417792264916992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26311811 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443949602738176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7069712701952Œ½g2– ØO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110095938 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424862002522624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _runtime‚5436.707504407 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443949619777536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚417792289820672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436485786847232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450793149368832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110105915 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443305240658992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6819453811760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6843529591296 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6843441654272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443949619777536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6843441654272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87050939 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843734952635026e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79791544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26311579 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚11599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87050939 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450793044392448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424861803411968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚450793149368832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚450793044392448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6843529591296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 _step‚939 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79791779 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87050941 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7069538494976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110095938 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6819367674670 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197146877 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417792264916992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26311811 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443949602738176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_4k¹XÁ pool.allocated‚ 7069712701952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53479965 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443305137482542 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚436485769807872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197156856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146«/ž‡Ã2— - _wandbruntime‚5437  _runtime‚5437§øpã&˜ ÝM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6844011013632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87058160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443986713702400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443986730741760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚417827176431616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197173372 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443986713702400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚450830829699584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110115210 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚443342280015962 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6820021133402 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424897301664256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197173372 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424897502864896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443342176839400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6819934996200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450830829699584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79798123 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87058160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26313893 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 % _timestamp‚1.7843734956037407e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197163379 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚5437.047850914 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6844098957824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87058162 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110105219 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197163379 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436522241843200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚11599 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064%K¾Iè&™ âM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6844098957824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87058162 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110105219 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197163379 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436522241843200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚11599 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚219 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6844011013632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87058160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443986713702400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443986730741760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚417827176431616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197173372 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443986713702400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚450830829699584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110115210 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚443342280015962 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6820021133402 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424897301664256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197173372 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201200640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424897502864896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443342176839400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6819934996200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450830829699584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79798123 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87058160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26313893 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 % _timestamp‚1.7843734956037407e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197163379 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 _runtime‚5437.047850914¬­j˽2š - _wandbruntime‚5437  _runtime‚5437OÔ › … %loss/total_loss‚8.429461508784192 epoch‚14 trainer/global_step‚11599 0loss/FRt1/assignment_loss‚1.6405108886280095 2loss/FBt2/assignment_loss‚0.015774525139614602 /loss/FRt2/detection_loss‚1.5422663419188833 /loss/FBt1/detection_loss‚0.6745484273292517 /loss/FBt2/detection_loss‚0.5193480763316028 2loss/SRqqt1/assignment_loss‚0.5884774683047402 3loss/SRqqt2/assignment_loss‚0.49064955156907625 1loss/SRqqt2/detection_loss‚0.7172887471897863 % _timestamp‚1.7843734956053493e+09 _runtime‚ 5437.0488927 _step‚941 0loss/FRt2/assignment_loss‚0.6820072603853126 4loss/FBt1/assignment_loss‚0.00034921997658481015 /loss/FRt1/detection_loss‚0.5938336419853374 0loss/SRqqt1/detection_loss‚0.964407360025992ó—óœ Š _step‚941 0loss/FRt2/assignment_loss‚0.6820072603853126 4loss/FBt1/assignment_loss‚0.00034921997658481015 /loss/FRt1/detection_loss‚0.5938336419853374 0loss/SRqqt1/detection_loss‚0.964407360025992 %loss/total_loss‚8.429461508784192 epoch‚14 trainer/global_step‚11599 0loss/FRt1/assignment_loss‚1.6405108886280095 2loss/FBt2/assignment_loss‚0.015774525139614602 /loss/FRt2/detection_loss‚1.5422663419188833 /loss/FBt1/detection_loss‚0.6745484273292517 /loss/FBt2/detection_loss‚0.5193480763316028 2loss/SRqqt1/assignment_loss‚0.5884774683047402 3loss/SRqqt2/assignment_loss‚0.49064955156907625 1loss/SRqqt2/detection_loss‚0.7172887471897863 % _timestamp‚1.7843734956053493e+09 _runtime‚ 5437.0488927­A±M‘ ‚(:† ÿÁíÒÔóªŠ gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚197.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.4266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýL·åž ‚(:Ú ÿÁíÒ•öû network.sent‚ 1.5788104e+07 disk.rbd1.in‚0( disk.nvme0c0n1.out‚13683.52294921875 disk.sda.in‚0.109375 disk.rbd1.out‚ 12.9140625# disk.nvme0c0n1.in‚ 7203.47265625 disk.sda2.out‚0$ memory_percent‚99.28312301635742 disk.drbd2165.in‚0 disk.sda1.out‚0# disk.dm-0.out‚13683.52294921875& proc.memory.availableMB‚ 117.453125# proc.memory.rssMB‚ 2704.20703125) proc.memory.percent‚16.50516986846924 disk.sda.out‚ 178.8359375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 178.8359375 disk.rbd0.out‚0 disk.sda3.out‚ 178.8359375) disk./.usagePercent‚39.81845773565473 disk.dm-1.in‚0.109375 proc.cpu.threads‚17 network.recv‚ 1.663471e+06 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 7203.47265625& disk.nvme0n1.out‚13683.52294921875 disk.dm-0.in‚ 7203.47265625 disk.sda3.in‚0.109375$ disk./.usageGB‚593.3237953186035 cpu‚26.835500383592162Á<'.2Ÿ - _wandbruntime‚5458  _runtime‚5458¨švÆ  À # _timestamp‚1.78437351692141e+09 _runtime‚5458.365229343 _step‚942 & lr-AdamW/pg1‚0.0006980062183117246 & lr-AdamW/pg2‚0.0006980062183117246 trainer/global_step‚11649i—T Ë¡ Å & lr-AdamW/pg1‚0.0006980062183117246 & lr-AdamW/pg2‚0.0006980062183117246 trainer/global_step‚11649 # _timestamp‚1.78437351692141e+09 _runtime‚5458.365229343 _step‚942®…ªÚ"2¢ - _wandbruntime‚5458  _runtime‚54580ìÑ(£ ÓO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6871941879296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6871941879296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452677197805056 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6872029816320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419537840648192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445805167988736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445805167988736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452677092828672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87411991 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445157137228650 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 trainer/global_step‚11649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26420647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419537865551872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚943 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26420412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 684510”õ± 4128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110571817 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445805150949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110571817 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445157240405100 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87411989 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚5458.366597585 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197983808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452677092828672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6847765654378 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024c+­ÈÞ'¤ ØO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80124577 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 % _timestamp‚1.7843735169225261e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87411989 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7099053315584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452677197805056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6872029816320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7099233814016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438309388613632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445805167988736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445805167988736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452677092828672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87411991 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445157137228650 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 trainer/global_step‚11649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26420647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419537865551872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚943 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26420412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205402112 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110571817 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445805150949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110571817 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445157240405100 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87411989 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚5458.366597585 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197983808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452677092828672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6847765654378 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6871941879296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6871941879296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452677197805056 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6872029816320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419537840648192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚11649 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6872511238656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445842278952960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843735172650356e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110581112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452714773152256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6872599182848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7099641482752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26422584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87419212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110581112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110571121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87419212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6848419113110 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452714878135808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53708649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26422827 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7099813585408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198000324 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80131479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426672590651904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110571121 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197990331 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445194176585508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445842261913600 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6872599182848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80131233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445842278952960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6848332975908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198000324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419572752162816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438345843609600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426672393645568 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445194279762070 _runtime‚ 5458.70915843 _step‚944#nÐ#:#§ áM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445194176585508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445842261913600 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6872599182848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80131233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445842278952960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6848332975908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198000324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419572752162816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438345843609600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426672393645568 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445194279762070 _runtime‚ 5458.70915843 _step‚944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438345860648960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445842261913600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6872511238656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53708652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 trainer/global_step‚11649 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 197006336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6872511238656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445842278952960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843735172650356e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110581112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452714773152256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6872599182848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7099641482752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26422584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87419212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110581112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110571121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87419212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6848419113110 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452714878135808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53708649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26422827 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7099813585408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198000324 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80131479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426672590651904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110571121 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197990331 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitÚiâ­tor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6900530041344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80457932 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 trainer/global_step‚11699 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87773039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6876249771176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447660699160576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111037719 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6900442104320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6876163634086 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87773039 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26529792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440132973340672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454561246241280 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454561141264896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428411883518464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440132990380032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447660716199936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111027742 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6900530041344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80457932 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 trainer/global_step‚11699 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87773039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6876249771176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447660699160576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111037719 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6900442104320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6876163634086 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87773039 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26529792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440132973340672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454561246241280 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454561141264896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428411883518464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440132990380032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447660716199936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 192819200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327³p.2µ - _wandbruntime‚5480  _runtime‚5480Øò˶ ÝM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421318327894016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7129056088576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6876730955616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198817283 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚214 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454598821588480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26531702 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80464765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26531913 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428447580988928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚5480.580737936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚211 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7129228191232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6901099407872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11699 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87780260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87780262 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111037023 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53932849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843735391366339e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198827276 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447697810124800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447697827164160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901011463680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚440169462415360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421318352797696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111047014 REDeviceStatsMonitor.on_trainâA_batch_end/active_bytes.small_pool.current‚87944192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447046279508178 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447697827164160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80464551 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447046176331616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198827276 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447697827164160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901011463680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚440169462415360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421318352797696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111047014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447046279508178 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447697827164160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80464551 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447046176331616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198827276 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421318327894016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7129056088576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6876730955616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198817283 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚214 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454598821588480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26531702 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80464765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26531913 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428447580988928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚5480.580737936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚211 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7129228191232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6901099407872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11699 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87780260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87780262 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111037023 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53932849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843735391366339e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198827276 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447697810124800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5501.891565302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111493644 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456445189701120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448861136720866 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111503621 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111493644 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80790187 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54152568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6904561613794 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448861239897316 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6904647750884 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 trainer/global_step‚11749 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54152565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449516247371776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80790423 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430187128743424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88134091 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423028992110592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6928942329344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199637712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88134089 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚449516264411136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449516247371776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26637622 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7157937522176 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6928942329344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7158111729152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7158111729152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5501.891565302 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111493644 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456445189701120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448861136720866 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111503621 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111493644 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80790187 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54152568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6904561613794 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448861239897316 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6904647750884 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 trainer/global_step‚11749 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54152565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449516247371776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80790423 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430187128743424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88134091 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199110656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423028992110592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6928942329344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199637712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88134089 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174206976 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚449516264411136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449516247371776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26637622 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7157937522176 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6928942329344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360·­tž2à - _wandbruntime‚5502  _runtime‚5502ÕpÄá&Ä ÛM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456482975008256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199654228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199644235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423063903625216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6905128935324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80796862 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111502925 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6929599632896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚456482870024704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6929511688704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54157052 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199644235 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚215 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430222626451968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449553358336000ƒ4×æ&Å àM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199644235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192812032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423063903625216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6905128935324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167908352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80796862 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111502925 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6929599632896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚456482870024704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6929511688704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54157052 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199644235 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚215 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430222626451968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449553358336000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456482975008256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199654228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346¸1í5Æ2Æ - _wandbruntime‚5502  _runtime‚5502–ïøì Ç † trainer/global_step‚11749 0loss/FRt1/assignment_loss‚1.5658800726556508 2loss/FBt2/assignment_loss‚0.004293782236781024 1loss/SRqqt2/detection_loss‚0.6515903385165247 2loss/FBt1/assignment_loss‚0.008331002161583001 /loss/FRt2/detection_loss‚1.3361293834805563 %loss/total_loss‚7.602611822188538 epoch‚14 % _timestamp‚1.7843735607843986e+09 _step‚953 0loss/FRt2/assignment_loss‚0.6681077329080496 2loss/SRqqt1/assignment_loss‚0.4961569432234862 1loss/SRqqt1/detection_loss‚0.6283439609859818 _runtime‚5502.227939589 3loss/SRqqt2/assignment_loss‚0.46453287888433825 /loss/FRt1/detection_loss‚0.5931345541668562 /loss/FBt1/detection_loss‚0.6272504274966195 /loss/FBt2/detection_loss‚0.5588607454721107ÅÄ1TÈ ‹ 0loss/FRt2/assignment_loss‚0.6681077329080496 2loss/SRqqt1/assignment_loss‚0.4961569432234862 1loss/SRqqt1/detection_loss‚0.6283439609859818 _runtime‚5502.227939589 3loss/SRqqt2/assignment_loss‚0.46453287888433825 /loss/FRt1/detection_loss‚0.5931345541668562 /loss/FBt1/detection_loss‚0.6272504274966195 /loss/FBt2/detection_loss‚0.5588607454721107 trainer/global_step‚11749 0loss/FRt1/assignment_loss‚1.5658800726556508 2loss/FBt2/assignment_loss‚0.004293782236781024 1loss/SRqqt2/detection_loss‚0.6515903385165247 2loss/FBt1/assignment_loss‚0.008331002161583001 /loss/FRt2/detection_loss‚1.3361293834805563 %loss/total_loss‚7.602611822188538 epoch‚14 % _timestamp‚1.7843735607843986e+09 _step‚953¹O:°œ‘É ‚(:† »ÂíÒ îÈŠ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚182.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$z »ßÊ ‚(:Ô »ÂíÒñ‹”$ memory_percent‚99.28276538848877 cpu‚26.82906848601668 disk.sda3.out‚ 179.68359375 disk.sda.out‚ 179.68359375" disk.nvme0c0n1.in‚ 7246.6171875( proc.memory.availableMB‚ 117.51171875 network.recv‚ 1.679948e+06 disk.sda3.in‚0.109375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 7246.6171875$ disk.nvme0n1.out‚13703.326171875# disk./.usageGB‚593.325080871582 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2704.20703125) disk./.usagePercent‚39.81854401019377 disk.rbd1.in‚0 disk.rbd1.out‚ 13.0390625& disk.nvme0c0n1.out‚13703.326171875 disk.dm-0.in‚ 7246.6171875 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.50516986846924 network.sent‚ 1.5981024e+07 disk.sda2.in‚0 disk.dm-1.in‚0.109375! disk.dm-0.out‚13703.326171875 disk.sda.in‚0.109375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 179.683593755ÝÑ‘Ë ‚(:† ÊÂíÒü†¯Š gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™UÔ3ßÌ ‚(:Ô ÊÂíÒè½çŒ disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 13.1015625$ memory_percent‚99.28462505340576 disk.nvme0n1.in‚ 7246.6171875 disk.rbd0.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚39.8185524584424 disk.sda1.in‚0) proc.memory.percent‚16.50516986846924 cpu‚26.82759640463979 disk.sda3.out‚ 179.9296875 disk.sda.out‚ 179.9296875( proc.memory.availableMB‚ 117.20703125# proc.memory.rssMB‚ 2704.20703125$ disk./.usageGB‚593.3252067565918' disk.nvme0c0n1.out‚13709.5341796875% disk.nvme0n1.out‚13709.5341796875 disk.sda1.out‚0 disk.dm-1.out‚ 179.9296875 network.recv‚ 1.683924e+06 disk.sda2.in‚0 disk.dm-1.in‚0.109375 disk.sda.in‚0.109375 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 7246.6171875 network.sent‚ 1.6043638e+07 disk.rbd0.out‚0 disk.dm-0.in‚ 7246.6171875" disk.dm-0.out‚13709.5341796875 disk.sda3.in‚0.109375tŸÙé2Í - _wandbruntime‚5523  _runtime‚5523ꪃdÈÎ Â & lr-AdamW/pg1‚0.0006941008876514597 & lr-AdamW/pg2‚0.0006941008876514597 trainer/global_step‚11799 % _timestamp‚1.7843735824090216e+09 _runtime‚5523.852845833 _step‚954ïò"ÍÏ Ç _step‚954 & lr-AdamW/pg1‚0.0006941008876514597 & lr-AdamW/pg2‚0.0006941008876514597 trainer/global_step‚11799 % _timestamp‚1.7843735824090216e+09 _runtime‚5523.852845833º‹·*2Ð - _wandbruntime‚5523  _runtime‚5523õmk=´ Ñ ÓO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458329343113728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88495141 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451371795582976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54376768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚450713136466974 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚424774592745472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚424774567841792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451371812622336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚955 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8849514®ÿ·%1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111959546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚5523.854135198 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458329343113728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26746028 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26746250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6957530491392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6957442554368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81122793 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450713239643424 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111969523 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431962166208512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443780176873472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88495139 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6957442554368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458329238137344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7187390867456 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6957530491392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54376765 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7187573463040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451371795582976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111969523 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200454685 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88495139 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111959546 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81123018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6933045730592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200464664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458329238137344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443780193912832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090_@´Þ'Ò ØO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 _step‚955 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88495141 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111959546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 _runtime‚5523.854135198 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458329343113728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26746028 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26746250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6957530491392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6957442554368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81122793 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚450713239643424 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111969523 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431962166208512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443780176873472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88495139 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6957442554368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458329238137344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7187390867456 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6957530491392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54376765 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7187573463040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451371795582976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207499264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111969523 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200454685 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88495139 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111959546 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81123018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182595584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6933045730592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200464664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458329238137344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443780193912832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458329343113728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88495141 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451371795582976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54376768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚450713136466974 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚424774592745472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚424774567841792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451371812622336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146»Ïó“:2Ó - _wandbruntime‚5524  _runtime‚5524ðÚÕã&Ô ÝM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443816665948160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451408923586560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458366918460928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88502362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7188155585536 % _timestamp‚1.7843735827508762e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6958099857920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26748333 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431997659845632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54381249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7187977191424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6958011913728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450750279000394 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443816648908800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450750175823832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚424809504260096 _step‚956 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111968827 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26748105 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200471187 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54381252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚424809479356416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200471187 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚11799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88502360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111978818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5524.194950078 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6958011913728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458367023444480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451408923586560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111968827 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88502360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200481180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81129354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111978818 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451408906547200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200481180 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81129585 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431997456547840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458367023444480 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6933613052234¹î[8ÅÕ âM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458367023444480 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6933613052234 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443816665948160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451408923586560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458366918460928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88502362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7188155585536 % _timestamp‚1.7843735827508762e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6958099857920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26748333 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431997659845632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54381249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178394112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7187977191424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6958011913728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450750279000394 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443816648908800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450750175823832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚424809504260096 _step‚956 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111968827 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26748105 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607Åú æ# VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200471187 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54381252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203297792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚424809479356416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200471187 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 trainer/global_step‚11799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88502360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111978818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5524.194950078 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6958011913728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458367023444480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451408923586560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111968827 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88502360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200481180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81129354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111978818 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451408906547200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200481180 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81129585 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431997456547840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473¼Sj8å2Ö - _wandbruntime‚5524  _runtime‚5524%0 × „ 2loss/SRqqt1/assignment_loss‚0.6371196912329714 1loss/SRqqt1/detection_loss‚0.8376265215692806 .loss/FBt2/detection_loss‚0.375695002892905 epoch‚14 _runtime‚5524.195996744 0loss/FRt2/assignment_loss‚0.6119434222871182 0loss/FBt2/assignment_loss‚0.0158955608831064 0loss/FBt1/detection_loss‚0.43941723369061947 % _timestamp‚1.7843735827524524e+09 4loss/FBt1/assignment_loss‚0.00012478219461968366 .loss/FRt1/detection_loss‚0.611676375932233 /loss/FRt2/detection_loss‚1.4045086271773257 %loss/total_loss‚7.604474666156979 trainer/global_step‚11799 0loss/FRt1/assignment_loss‚1.6397306662024433 3loss/SRqqt2/assignment_loss‚0.48594096526776775 0loss/SRqqt2/detection_loss‚0.544795816826589 _step‚957C[ðØ ‰ 0loss/FRt2/assignment_loss‚0.6119434222871182 0loss/FBt2/assignment_loss‚0.0158955608831064 0loss/FBt1/detection_loss‚0.43941723369061947 % _timestamp‚1.7843735827524524e+09 4loss/FBt1/assignment_loss‚0.00012478219461968366 .loss/FRt1/detection_loss‚0.611676375932233 /loss/FRt2/detection_loss‚1.4045086271773257 %loss/total_loss‚7.604474666156979 trainer/global_step‚11799 0loss/FRt1/assignment_loss‚1.6397306662024433 3loss/SRqqt2/assignment_loss‚0.48594096526776775 0loss/SRqqt2/detection_loss‚0.544795816826589 _step‚957 2loss/SRqqt1/assignment_loss‚0.6371196912329714 1loss/SRqqt1/detection_loss‚0.8376265215692806 .loss/FBt2/detection_loss‚0.375695002892905 epoch‚14 _runtime‚5524.195996744½9,‘Ù ‚(:† ÙÂíÒ¨ˆ»Š gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚125.789' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úºèåÚ ‚(:Ú ÙÂíÒ–é¿ disk.dm-0.in‚ 7246.6171875 disk.sda3.in‚0.109375 disk.sda.out‚ 180.3046875 disk.rbd0.out‚0) proc.memory.percent‚16.50516986846924 network.recv‚ 1.686766e+06* disk./.usagePercent‚39.818541962133494# disk.dm-0.out‚13712.74267578125 disk.rbd1.in‚0$ disk./.usageGB‚593.3250503540039( disk.nvme0c0n1.out‚13712.74267578125 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 180.3046875 proc.cpu.threads‚17& disk.nvme0n1.out‚13712.74267578125 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2704.20703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.6047681e+07 disk.sda.in‚0.109375 disk.rbd0.in‚0 cpu‚26.826032152267945 disk.sda3.out‚ 180.3046875 disk.rbd1.out‚ 13.1015625 disk.nvme0n1.in‚ 7246.6171875 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 117.67578125" disk.nvme0c0n1.in‚ 7246.6171875$ memory_percent‚99.28176403045654A7s)2Û - _wandbruntime‚5545  _runtime‚5545oAæãÆÜ À _step‚958 & lr-AdamW/pg1‚0.0006927904314387559 & lr-AdamW/pg2‚0.0006927904314387559 trainer/global_step‚11849 # _timestamp‚1.78437360442309e+09 _runtime‚5545.866853266r×ËÝ Å _runtime‚5545.866853266 _step‚958 & lr-AdamW/pg1‚0.0006927904314387559 & lr-AdamW/pg2‚0.0006927904314387559 trainer/global_step‚11849 # _timestamp‚1.78437360442309e+09¾Ý®Øö2Þ - _wandbruntime‚5545  _runtime‚5545uU« Ù'ß ÓO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚445603795679232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843736044241295e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201291616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460213286573568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚452565136213082 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6986030716416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26855276 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452565239389532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453227360833536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112435425 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112425448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7216854443008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6985942779392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453227360833536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88856189 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453227343794176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112435425 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460213286573568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6985942779392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6961357573210 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453227343794176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54600968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7217024455680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445603778639872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201281637 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460213391549952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81456241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426520168476672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433736998016000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 _runtime‚5545.868086323 _step‚959 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433737192932352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6986030716416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81456477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54600965 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88856191 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460213391549952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26855509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6961443710300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11849 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88856191 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88856189 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚13467– UÞ'à ØO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112435425 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112425448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7216854443008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6985942779392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453227360833536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88856189 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453227343794176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112435425 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194916352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460213286573568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6985942779392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6961357573210 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453227343794176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54600968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7217024455680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445603778639872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201281637 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460213391549952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81456241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426520168476672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433736998016000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 _runtime‚5545.868086323 _step‚959 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433737192932352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6986030716416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81456477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54600965 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88856191 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170012672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460213391549952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26855509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6961443710300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11849 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88856191 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88856189 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚445603795679232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843736044241295e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201291616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460213286573568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚452565136213082 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6986030716416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26855276 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452565239389532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453227360833536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¿Ž|D 2á - _wandbruntime‚5546  _runtime‚5546Æ«)Üâ ÝM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88863412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26857426 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201308132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460251071880704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6986600082944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6986512138752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201308132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26857660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433772494846976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88863412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88863410 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112444720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88863410 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112434729 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6Jý›015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54605452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112434729 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81462875 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460251071880704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6962011031942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81463112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7217611862016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452602175569940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚452602278746502 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445640250675200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453264454758400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460250966897152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7217439759360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _runtime‚5546.207097876 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81462875 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460251071880704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6962011031942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81463112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7217611862016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452602175569940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚452602278746502 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445640250675200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453264454758400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460250966897152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7217439759360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 _runtime‚5546.207097876 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201308132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460251071880704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6986600082944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6986512138752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201308132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26857660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433772494846976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88863412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88863410 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112444720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88863410 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112434729 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172102656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚327 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237ÀåïÖ2ä - _wandbruntime‚5546  _runtime‚5546{£Ë4 å … 0loss/FRt2/assignment_loss‚0.7073769362319803 2loss/SRqqt2/assignment_loss‚0.5028572135243211 1loss/SRqqt2/detection_loss‚0.5915424407544841 0loss/FBt2/detection_loss‚0.45227272290593157 epoch‚14 _step‚961 0loss/FRt1/assignment_loss‚1.5726277361990828 1loss/SRqqt1/assignment_loss‚0.630423004196066 _runtime‚5546.208337246 3loss/FBt1/assignment_loss‚0.0007922274646753439 2loss/FBt2/assignment_loss‚0.007470338906560524 /loss/FRt1/detection_loss‚0.5476876512231105 /loss/FRt2/detection_loss‚1.4674820059543134 1loss/SRqqt1/detection_loss‚0.7808268894931596 /loss/FBt1/detection_loss‚0.7271096044830208 %loss/total_loss‚7.988468771336706 trainer/global_step‚11849 $ _timestamp‚1.784373604764798e+09í©WÐæ Š 0loss/FRt2/assignment_loss‚0.7073769362319803 2loss/SRqqt2/assignment_loss‚0.5028572135243211 1loss/SRqqt2/detection_loss‚0.5915424407544841 0loss/FBt2/detection_loss‚0.45227272290593157 epoch‚14 _step‚961 0loss/FRt1/assignment_loss‚1.5726277361990828 1loss/SRqqt1/assignment_loss‚0.630423004196066 _runtime‚5546.208337246 3loss/FBt1/assignment_loss‚0.0007922274646753439 2loss/FBt2/assignment_loss‚0.007470338906560524 /loss/FRt1/detection_loss‚0.5476876512231105 /loss/FRt2/detection_loss‚1.4674820059543134 1loss/SRqqt1/detection_loss‚0.7808268894931596 /loss/FBt1/detection_loss‚0.7271096044830208 %loss/total_loss‚7.988468771336706 trainer/global_step‚11849 $ _timestamp‚1.784373604764798e+09ÁMʬýç ‚(:… èÂíÒ¯Š gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚182.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{AÔÝè ‚(:Ò èÂíÒäãéŒ( disk./.usagePercent‚39.8185539944876 disk.rbd1.in‚0 disk.nvme0n1.in‚ 7388.7734375 disk.sda1.in‚0" disk.dm-0.out‚13718.4814453125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 cpu‚26.82440004887792 proc.cpu.threads‚17# proc.memory.availableMB‚116.375 disk.sda3.in‚0.109375 disk.sda.out‚ 180.75390625 disk.drbd2165.out‚0% disk.nvme0n1.out‚13718.4814453125# proc.memory.rssMB‚ 2704.20703125 disk.sda.in‚0.109375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 180.75390625" disk.nvme0c0n1.in‚ 7388.7734375' disk.nvme0c0n1.out‚13718.4814453125 network.recv‚ 1.691224e+06 disk.sda2.in‚0 disk.dm-0.in‚ 7388.7734375) proc.memory.percent‚16.50516986846924$ memory_percent‚99.28970336914062 disk.rbd1.out‚ 13.1640625 network.sent‚ 1.6110462e+07$ disk./.usageGB‚593.3252296447754 disk.sda3.out‚ 180.75390625 disk.drbd2165.in‚0Ÿ—Êé ‚(:… ÷ÂíÒÛÌ‹ gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚183.695' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.739 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M¨Äèê ‚(:Ý ÷ÂíÒªñ³Œ disk.sda2.out‚0 disk.dm-0.in‚ 7415.84765625 network.recv‚ 1.695674e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 180.98046875 disk.sda3.out‚ 180.98046875) disk./.usagePercent‚39.81855553053281# disk.nvme0c0n1.in‚ 7415.84765625( proc.memory.availableMB‚ 116.38671875 disk.sda.in‚0.109375 disk.sda.out‚ 180.98046875 cpu‚26.822953038705652 disk.rbd0.out‚0 disk.sda3.in‚0.109375 disk.rbd1.in‚0 disk.rbd1.out‚ 13.1640625& disk.nvme0n1.out‚13721.43701171875 proc.cpu.threads‚17 network.sent‚ 1.6173203e+07# disk.dm-0.out‚13721.43701171875( disk.nvme0c0n1.out‚13721.43701171875# proc.memory.rssMB‚ 2704.20703125) proc.memory.percent‚16.50516986846924 disk.dm-1.in‚0.109375 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 7415.84765625# memory_percent‚99.2896318435669# disk./.usageGB‚593.325252532959 disk.sda1.in‚0Ø!ð©2ë - _wandbruntime‚5567  _runtime‚5567ÆÞ ÝÅì ¿ trainer/global_step‚11899 $ _timestamp‚1.784373626057235e+09 _runtime‚5567.500990989 _step‚962 % lr-AdamW/pg1‚0.000691475670487978 % lr-AdamW/pg2‚0.000691475670487978ྔÊí Ä trainer/global_step‚11899 $ _timestamp‚1.784373626057235e+09 _runtime‚5567.500990989 _step‚962 % lr-AdamW/pg1‚0.000691475670487978 % lr-AdamW/pg2‚0.000691475670487978ÂD¸[v2î - _wandbruntime‚5567  _runtime‚5567¿cQÙ'ï ÓO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89217239 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462097439986176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚11899 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462097439986176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202108589 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202118568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462097335009792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚963 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455082909044736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6989755552918 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428265744207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7246329852928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447427397445632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89217241 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462097335009792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435512241976320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 trainer/global_step‚11899 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462097439986176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202108589 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202118568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462097335009792 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚963 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455082909044736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6989755552918 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428265744207872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7246329852928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447427397445632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327ÃÜÈiU2ñ - _wandbruntime‚5567  _runtime‚5567MÑõ@â&ò ÜM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184685568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 _step‚964 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112910622 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455120002969600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _runtime‚5567.838494839 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462135120316928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7015012363776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7246909762560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428300630818816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81795719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435547750170624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚454454278492610 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462135120316928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447463869480960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202135084 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455120020008960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26966070 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89224462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112900631 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202135084 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447463852441600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89224460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11899 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7015012363776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202125091 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112910622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462135015333376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202135084 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447463852441600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89224460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚221 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11899 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7015012363776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202125091 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112910622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462135015333376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 306184192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455120002969600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 300376064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚146 _runtime‚5567.838494839 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462135120316928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7015012363776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7246909762560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209589248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428300630818816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81795719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435547750170624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚454454278492610 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462135120316928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447463869480960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202135084 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455120020008960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26966070 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89224462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Ä@QŠ2ô - _wandbruntime‚5567  _runtime‚55675þ°š õ ƒ 4loss/FBt2/assignment_loss‚0.00024869798488540363 1loss/SRqqt1/detection_loss‚0.8105220500126783 .loss/FBt2/detection_loss‚0.480382711160928 epoch‚14 _step‚965 2loss/SRqqt1/assignment_loss‚0.5202574428962274 0loss/FRt1/assignment_loss‚1.6342677787179127 0loss/FRt2/assignment_loss‚0.5028206513478206 2loss/FBt1/assignment_loss‚0.007882252842226425 /loss/FRt2/detection_loss‚1.5168690725301321 1loss/SRqqt2/detection_loss‚0.6427427269834912 /loss/FBt1/detection_loss‚0.6383858176845091 trainer/global_step‚11899 $ _timestamp‚1.784373626396388e+09 1loss/SRqqt2/assignment_loss‚0.533465012414098 /loss/FRt1/detection_loss‚0.5830116958495211 $loss/total_loss‚7.87085591042443 _runtime‚5567.839933073ñr[ªö ˆ 1loss/SRqqt1/detection_loss‚0.8105220500126783 .loss/FBt2/detection_loss‚0.480382711160928 epoch‚14 _step‚965 2loss/SRqqt1/assignment_loss‚0.5202574428962274 0loss/FRt1/assignment_loss‚1.6342677787179127 0loss/FRt2/assignment_loss‚0.5028206513478206 2loss/FBt1/assignment_loss‚0.007882252842226425 /loss/FRt2/detection_loss‚1.5168690725301321 1loss/SRqqt2/detection_loss‚0.6427427269834912 /loss/FBt1/detection_loss‚0.6383858176845091 trainer/global_step‚11899 $ _timestamp‚1.784373626396388e+09 1loss/SRqqt2/assignment_loss‚0.533465012414098 /loss/FRt1/detection_loss‚0.5830116958495211 $loss/total_loss‚7.87085591042443 _runtime‚5567.839933073 4loss/FBt2/assignment_loss‚0.00024869798488540363Å€ ¾[÷ ‚(:… †ÃíÒ÷Í Š gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚170.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.089 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðàçø ‚(:Ü †ÃíÒ¯ÀóŒ disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 13.2265625( proc.memory.availableMB‚ 116.15234375 cpu‚26.821370912299034 network.recv‚ 1.699136e+06 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚13727.32373046875& disk.nvme0n1.out‚13727.32373046875 network.sent‚ 1.6178202e+07* disk./.usagePercent‚39.818557322585555 disk.rbd0.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 7419.45703125# proc.memory.rssMB‚ 2704.20703125 disk.dm-1.in‚0.109375 disk.sda.out‚ 181.2734375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7419.45703125) proc.memory.percent‚16.50516986846924 disk.sda3.in‚0.109375 disk.sda3.out‚ 181.2734375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚13727.32373046875# memory_percent‚99.2910623550415$ disk./.usageGB‚593.3252792358398 disk.dm-1.out‚ 181.2734375 disk.dm-0.in‚ 7419.45703125 disk.sda.in‚0.109375œâ‚G2ù - _wandbruntime‚5589  _runtime‚55893øøéÈú  _step‚966 & lr-AdamW/pg1‚0.0006901566258576907 & lr-AdamW/pg2‚0.0006901566258576907 trainer/global_step‚11949 % _timestamp‚1.7843736479580195e+09 _runtime‚5589.401915679›gÿÍû Ç & lr-AdamW/pg2‚0.0006901566258576907 trainer/global_step‚11949 % _timestamp‚1.7843736479580195e+09 _runtime‚5589.401915679 _step‚966 & lr-AdamW/pg1‚0.0006901566258576907Æ{)˜2ü - _wandbruntime‚5589  _runtime‚5589 äN¶ý ÓO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚463981383446016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.fr½÷ñ#%eed‚456269135705298 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449250982172672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚463981488422400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27073182 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456269238881748 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚11949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202945520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55049365 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113367229 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7043031166464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚473 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚463981383446016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89578291 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113367229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7043031166464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7275810935808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449250999212032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89578289 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437287300887552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113357252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456938457255936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27073403 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚463981488422400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7042943229440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7042943229440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55049368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456938457255936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82122771 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82122547 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7275980948480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚463981383446016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89578291 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113367229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7043031166464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7275810935808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449250999212032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89578289 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437287300887552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113357252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 _step‚967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456938457255936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27073403 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚463981488422400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7042943229440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7042943229440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55049368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456938457255936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82122771 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82122547 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7275980948480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104976384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456269135705298 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449250982172672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚463981488422400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27073182 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456269238881748 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚146 trainer/global_step‚11949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202945520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55049365Çt °Ò2ÿ - _wandbruntime‚5589  _runtime‚5589žÇéSã&€!ÝM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449287471247360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449287454208000 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113376524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7276399037440 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464019168753152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464019168753152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82129153 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7018720854156 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202962036 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456306278238718 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _runtime‚5589.742819532 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7043600532992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456306175062156 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89585510 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456975568220160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27075517 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27075304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55053852 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202952043 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113366533 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7043600532992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456975551180800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82129369 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437322605587456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464019063769600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7043512588800 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113366533 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7018806991358 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473žÆýÇ !âM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449287454208000 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113376524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7276399037440 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464019168753152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464019168753152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82129153 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7018720854156 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6845104128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202962036 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456306278238718 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 _runtime‚5589.742819532 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7043600532992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚213 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456306175062156 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6538919936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89585510 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456975568220160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27075517 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27075304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55053852 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202952043 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113366533 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7043600532992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456975551180800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82129369 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437322605587456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464019063769600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7043512588800 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113366533 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6081670144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7018806991358 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚473 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5739898880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449287471247360ÈÛ ì2‚!- _wandbruntime‚5589  _runtime‚5589+4¦cƒ!ˆ 1loss/SRqqt1/assignment_loss‚0.574768108386082 3loss/FBt1/assignment_loss‚5.202937361680008e-06 /loss/FRt1/detection_loss‚0.5680670904805957 % _timestamp‚1.7843736483004794e+09 3loss/SRqqt2/assignment_loss‚0.42323564778319445 /loss/FRt2/detection_loss‚1.5196312344288108 3loss/FBt2/assignment_loss‚0.0015801534664713192 /loss/FBt1/detection_loss‚0.7711065517300192 %loss/total_loss‚7.996185524975914 _step‚969 0loss/FRt2/assignment_loss‚0.7350181081780681 1loss/SRqqt1/detection_loss‚0.7089253023396529 1loss/SRqqt2/detection_loss‚0.6863534962577089 0loss/FBt2/detection_loss‚0.38172233850480275 epoch‚14 trainer/global_step‚11949 _runtime‚5589.744010951 0loss/FRt1/assignment_loss‚1.6257722904831453o „! 3loss/FBt2/assignment_loss‚0.0015801534664713192 /loss/FBt1/detection_loss‚0.7711065517300192 %loss/total_loss‚7.996185524975914 _step‚969 0loss/FRt2/assignment_loss‚0.7350181081780681 1loss/SRqqt1/detection_loss‚0.7089253023396529 1loss/SRqqt2/detection_loss‚0.6863534962577089 0loss/FBt2/detection_loss‚0.38172233850480275 epoch‚14 trainer/global_step‚11949 _runtime‚5589.744010951 0loss/FRt1/assignment_loss‚1.6257722904831453 1loss/SRqqt1/assignment_loss‚0.574768108386082 3loss/FBt1/assignment_loss‚5.202937361680008e-06 /loss/FRt1/detection_loss‚0.5680670904805957 % _timestamp‚1.7843736483004794e+09 3loss/SRqqt2/assignment_loss‚0.42323564778319445 /loss/FRt2/detection_loss‚1.5196312344288108Éíב…!‚(:† •ÃíÒຠŠ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚30.0 gpu.0.powerWatts‚186.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Œyþé†!‚(:Þ •ÃíÒ×È´Œ disk.sda2.in‚0 disk.sda3.out‚ 181.40234375 disk.sda.in‚0.109375 disk.rbd1.in‚0 disk.rbd1.out‚ 13.2265625) proc.memory.percent‚16.50516986846924 disk.rbd0.out‚0 disk.sda3.in‚0.109375( disk.nvme0c0n1.out‚13731.90380859375 proc.cpu.threads‚17) disk./.usagePercent‚39.81856397878144 disk.sda1.out‚0 disk.dm-1.in‚0.109375 disk.dm-1.out‚ 181.40234375 disk.sda.out‚ 181.40234375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2704.20703125 cpu‚26.81980698150131 network.sent‚ 1.6240943e+07 disk.sda1.in‚0& disk.nvme0n1.out‚13731.90380859375 network.recv‚ 1.703751e+06 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7419.45703125( proc.memory.availableMB‚ 116.16015625$ disk./.usageGB‚593.3253784179688! disk.nvme0n1.in‚ 7419.45703125$ memory_percent‚99.29101467132568 disk.dm-0.in‚ 7419.45703125# disk.dm-0.out‚13731.90380859375ýr­¦‘‡!‚(:† ¤ÃíÒ늩Š gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½³¥J؈!‚(:Í ¤ÃíÒ±±ìŒ network.sent‚ 1.6303542e+07) disk./.usagePercent‚39.81855425049514 disk.sda3.out‚ 181.6953125 disk.sda.out‚ 181.6953125# disk.nvme0c0n1.out‚ 13740.453125! disk.nvme0n1.in‚ 7419.45703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2704.20703125 cpu‚26.818375264791207 network.recv‚ 1.707793e+06 disk.sda3.in‚0.109375 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 181.6953125$ memory_percent‚99.29144382476807$ disk./.usageGB‚593.3252334594727 disk.rbd1.out‚ 13.3515625! disk.nvme0n1.out‚ 13740.453125 disk.sda2.in‚0 disk.dm-0.in‚ 7419.45703125) proc.memory.percent‚16.50516986846924 disk.sda1.in‚0( proc.memory.availableMB‚ 116.08984375 disk.dm-1.in‚0.109375 disk.dm-0.out‚ 13740.453125 proc.cpu.threads‚17 disk.sda.in‚0.109375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7419.45703125‰k:Ä2‰!- _wandbruntime‚5611  _runtime‚5611ÎÐu}ÈŠ! _step‚970 & lr-AdamW/pg1‚0.0006888333186750713 & lr-AdamW/pg2‚0.0006888333186750713 trainer/global_step‚11999 % _timestamp‚1.7843736697193606e+09 _runtime‚5611.163125965 “‡lÍ‹!Ç % _timestamp‚1.7843736697193606e+09 _runtime‚5611.163125965 _step‚970 & lr-AdamW/pg1‚0.0006888333186750713 & lr-AdamW/pg2‚0.0006888333186750713 trainer/global_step‚11999Êò‘2Œ!- _wandbruntime‚5611  _runtime‚5611å)ß)Ù'!ÓO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465865431882240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458794005467136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182061 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439062367358464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7071443454464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458793988427776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458793988427776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439062161956352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89939339 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113823154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451074600978432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451074583939072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82455388 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55273565 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203762493 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚458121135451406 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465865431882240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7046551512334 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89939339 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55273568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7046637649424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89939341 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚971 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113833131 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113823154 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431756895670272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚458121238627856ÿ½zóÞ'Ž!ØO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182061 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439062367358464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7071443454464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458793988427776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458793988427776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439062161956352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89939339 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113823154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451074600978432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451074583939072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5812080640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82455388 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55273565 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203762493 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚458121135451406 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465865431882240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7046551512334 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89939339 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55273568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7046637649424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89939341 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚971 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113833131 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113823154 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431756895670272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180498432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚458121238627856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465865431882240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458794005467136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936Ëa…Q2!- _wandbruntime‚5611  _runtime‚5611V4T•W!ÜM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7072100758016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 cODeviceStatsMonitor.on_tb»œ>‹%rain_batch_end/inactive_split_bytes.large_pool.allocated‚431791807184896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458831099392000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7072100758016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113842426 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465903112205824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7306055136768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458158174808264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 $ _timestamp‚1.784373670061414e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465903217189376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113842426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458831099392000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82462279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7305885131264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7072012813824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203788988 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚11999 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465903217189376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113832435 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203788988 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89946560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439097862321664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451111055974400 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82462044 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89946560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203778995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431791782281216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5611.505573951x9iç&‘!áM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465903217189376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113842426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104983552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458831099392000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82462279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7305885131264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7072012813824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203788988 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6845104128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚11999 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465903217189376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113832435 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203788988 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89946560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439097862321664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451111055974400 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82462044 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89946560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203778995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5812080640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431791782281216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5611.505573951 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7072100758016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚431791807184896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458831099392000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7072100758016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113842426 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465903112205824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7306055136768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458158174808264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 $ _timestamp‚1.784373670061414e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚146Ìž­û·2’!- _wandbruntime‚5611  _runtime‚5611vßV± “!† 0loss/FRt2/assignment_loss‚0.5244055677897966 3loss/FBt1/assignment_loss‚0.0004503590672120102 0loss/FBt2/detection_loss‚0.23202074748148924 %loss/total_loss‚7.174872905861645 trainer/global_step‚11999 0loss/FRt1/assignment_loss‚1.5955201073167025 2loss/SRqqt1/assignment_loss‚0.5530639137187968 3loss/SRqqt2/assignment_loss‚0.45352745451478443 /loss/FBt1/detection_loss‚0.2882533134892583 epoch‚14 2loss/FBt2/assignment_loss‚0.011561076025836356 /loss/FRt1/detection_loss‚0.5912129434086782 -loss/FRt2/detection_loss‚1.49096397532328 1loss/SRqqt2/detection_loss‚0.6511822381230356 % _timestamp‚1.7843736700632558e+09 _runtime‚5611.506796378 1loss/SRqqt1/detection_loss‚0.7827112096027754 _step‚973®‡”!‹ 0loss/FRt2/assignment_loss‚0.5244055677897966 3loss/FBt1/assignment_loss‚0.0004503590672120102 0loss/FBt2/detection_loss‚0.23202074748148924 %loss/total_loss‚7.174872905861645 trainer/global_step‚11999 0loss/FRt1/assignment_loss‚1.5955201073167025 2loss/SRqqt1/assignment_loss‚0.5530639137187968 3loss/SRqqt2/assignment_loss‚0.45352745451478443 /loss/FBt1/detection_loss‚0.2882533134892583 epoch‚14 2loss/FBt2/assignment_loss‚0.011561076025836356 /loss/FRt1/detection_loss‚0.5912129434086782 -loss/FRt2/detection_loss‚1.49096397532328 1loss/SRqqt2/detection_loss‚0.6511822381230356 % _timestamp‚1.7843736700632558e+09 _runtime‚5611.506796378 1loss/SRqqt1/detection_loss‚0.7827112096027754 _step‚973Íü­úœ•!‚(:‘ ³ÃíÒÈžœŠ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚29.0 gpu.0.powerWatts‚116.621' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.324199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hº¯ñã–!‚(:Ø ³ÃíÒû禌 disk.sda2.in‚0 disk.dm-1.out‚ 181.97265625 disk.sda.out‚ 181.97265625' proc.memory.availableMB‚ 113.7265625 proc.cpu.threads‚17# disk.dm-0.out‚13744.08447265625$ memory_percent‚99.30586814880371 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.109375 disk.rbd1.out‚ 13.3515625" proc.memory.rssMB‚ 2706.6640625 disk.rbd1.in‚0 network.recv‚ 1.71156e+06$ disk./.usageGB‚593.3253440856934 disk.rbd0.in‚0 disk.sda3.in‚0.109375& disk.nvme0n1.out‚13744.08447265625! disk.nvme0c0n1.in‚ 7567.421875 cpu‚26.822975726110304 disk.sda.in‚0.109375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚13744.08447265625 disk.drbd2165.in‚0* proc.memory.percent‚16.520166397094727 disk.sda3.out‚ 181.97265625* disk./.usagePercent‚39.818561674713635 disk.nvme0n1.in‚ 7567.421875 disk.sda1.in‚0 disk.dm-0.in‚ 7567.421875 network.sent‚ 1.6308612e+07 disk.rbd0.out‚0­Â?2—!- _wandbruntime‚5638  _runtime‚5638¾•³O'˜!—{ @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10781250149011612 /particle/accuracy_4_of_4‚0.9292868375778198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7409993410110474 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12959319353103638 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7996699213981628 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_3‚0.06735718995332718 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6889706254005432 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4350000023841858 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.694135844707489 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15680524706840515 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7448225021362305 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7405356168746948 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4635888934135437 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6307256817817688 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975398182868958 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.658234179019928 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7624326348304749 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028921933844685555 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984780550003052 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2867680490016937 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6563870310783386 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7980164289474487 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_2_of_3‚0.5640572309494019 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6632817387580872 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43560609221458435 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1439816951751709 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8685909509658813 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14405325055122375 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15710604190826416 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974288940429688 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6461665034294128 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8262206315994263 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24744930863380432 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1523064225912094 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5241987109184265 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6663914918899536 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44937723875045776 _runtime‚5638.852939702 )loss/val_total_loss‚7.814931345135595 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12243727594614029 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03939121216535568 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_1_of_3‚0.9332065582275391 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015939811244606972 *jet/accuracy_1_of_4‚0.9629793763160706 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925132989883423 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6616480350494385 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10808666795492172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7680262327194214 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.053371861577034 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7635963559150696 jet/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04681244492530823 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28577694296836853 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009091528132557869 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945001006126404 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4535811245441437 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5286099314689636 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.35456642508506775 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8870176672935486 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49463048577308655 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16764412820339203 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995811939239502 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6692588925361633 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1301657110452652 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09815303236246109 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03483603522181511 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5909551978111267 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *particle/sensitivity‚0.625716507434845 +particle/specificity‚0.9123103022575378 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2430555373430252 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7071062922477722 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6892794966697693 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_5_of_5‚0 epoch‚14 *jet/accuracy_2_of_4‚0.7688601613044739 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 _step‚974 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7434037923812866 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13848674297332764 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.704541802406311 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962725043296814 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6632817387580872 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015986744314432144 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1442614495754242 *jet/accuracy_1_of_1‚0.3064437806606293 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21547581255435944 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4810166656970978 +jet/accuracy_4_of_4‚0.01230158656835556 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6940169930458069 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5959392189979553 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7817741632461548 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7846598029136658 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1439012736082077 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22161449491977692 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11427050828933716 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03483603522181511 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6980513334274292 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4856770634651184 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8262206315994263 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.767521321773529 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16405357420444489 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7635963559150696 =&Purity/*FRt*SRqqt*FBt/event_propoÓ0 INrtion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39113008975982666 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_2_of_2‚0.15294022858142853 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987118124961853 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5967743992805481 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12471529096364975 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8730583190917969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875355958938599 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028535667806863785 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5792506337165833 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46525028347969055 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11083953827619553 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18558137118816376 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7342538237571716 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8507643342018127 particle/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04661183804273605 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 particle/accuracy_1_of_4‚1 % _timestamp‚1.7843736974084878e+09 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.053061217069625854 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.760463535785675 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6640799641609192 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 'particle/f_score‚0.6429371237754822 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.800000011920929 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6967231631278992 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 )jet/accuracy_1_of_2‚0.778588056564331 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 jet/accuracy_3_of_5‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 particle/accuracy_1_of_2‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6871918439865112 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08806570619344711 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13526496291160583 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38817858695983887 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899723529815674 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 trainer/global_step‚12014 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15781262516975403 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9299271702766418 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7401547431945801 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1439816951751709 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7562581300735474 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21750333905220032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7464038729667664 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 particle/accuracy_1_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4135802090167999 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7634003758430481 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333134651184 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996954619884491 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14286115765571594 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7335102558135986 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0àlâ¢=™!œ{ ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7624326348304749 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028921933844685555 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984780550003052 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2867680490016937 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6563870310783386 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7980164289474487 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_2_of_3‚0.5640572309494019 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6632817387580872 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.43560609221458435 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1439816951751709 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8685909509658813 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14405325055122375 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15710604190826416 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974288940429688 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6461665034294128 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8262206315994263 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24744930863380432 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1523064225912094 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5241987109184265 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6663914918899536 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44937723875045776 _runtime‚5638.852939702 )loss/val_total_loss‚7.814931345135595 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12243727594614029 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03939121216535568 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_1_of_3‚0.9332065582275391 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015939811244606972 *jet/accuracy_1_of_4‚0.9629793763160706 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925132989883423 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6616480350494385 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10808666795492172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7680262327194214 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.053371861577034 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7635963559150696 jet/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04681244492530823 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28577694296836853 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009091528132557869 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945001006126404 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4535811245441437 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5286099314689636 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.35456642508506775 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8870176672935486 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49463048577308655 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16764412820339203 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995811939239502 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6692588925361633 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1301657110452652 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09815303236246109 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03483603522181511 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5909551978111267 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *particle/sensitivity‚0.625716507434845 +particle/specificity‚0.9123103022575378 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2430555373430252 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7071062922477722 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6892794966697693 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_5_of_5‚0 epoch‚14 *jet/accuracy_2_of_4‚0.7688601613044739 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 _step‚974 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7434037923812866 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13848674297332764 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.704541802406311 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962725043296814 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6632817387580872 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015986744314432144 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1442614495754242 *jet/accuracy_1_of_1‚0.3064437806606293 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21547581255435944 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4810166656970978 +jet/accuracy_4_of_4‚0.01230158656835556 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6940169930458069 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5959392189979553 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7817741632461548 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7846598029136658 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1439012736082077 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22161449491977692 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11427050828933716 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03483603522181511 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6980513334274292 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4856770634651184 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8262206315994263 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.767521321773529 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16405357420444489 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7635963559150696 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39113008975982666 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_2_of_2‚0.15294022858142853 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987118124961853 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5967743992805481 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12471529096364975 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8730583190917969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875355958938599 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028535667806863785 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5792506337165833 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46525028347969055 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11083953827619553 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18558137118816376 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7342538237571716 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8507643342018127 particle/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04661183804273605 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 particle/accuracy_1_of_4‚1 % _timestamp‚1.7843736974084878e+09 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.053061217069625854 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.760463535785675 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6640799641609192 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 'particle/f_score‚0.6429371237754822 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.800000011920929 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6967231631278992 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 )jet/accuracy_1_of_2‚0.778588056564331 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 jet/accuracy_3_of_5‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 particle/accuracy_1_of_2‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6871918439865112 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08806570619344711 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13526496291160583 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38817858695983887 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899723529815674 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 trainer/global_step‚12014 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15781262516975403 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9299271702766418 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7401547431945801 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1439816951751709 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7562581300735474 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21750333905220032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7464038729667664 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 particle/accuracy_1_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4135802090167999 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7634003758430481 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8333333134651184 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996954619884491 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14286115765571594 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7335102558135986 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10781250149011612 /particle/accuracy_4_of_4‚0.9292868375778198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7409993410110474 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12959319353103638 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7996699213981628 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_3‚0.06735718995332718 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6889706254005432 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4350000023841858 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.694135844707489 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15680524706840515 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7448225021362305 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7405356168746948 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4635888934135437 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6307256817817688 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975398182868958 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.658234179019928 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032Î* yjš!‚(:„ ÂÃíÒ²†ÊŠ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚28.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÒw*â›!‚(:× ÂÃíÒǪÑ disk.dm-1.in‚0.109375# disk.dm-0.out‚13749.91943359375' proc.memory.availableMB‚ 88.19140625$ disk./.usageGB‚593.3413124084473 disk.sda.in‚0.109375 disk.sda.out‚ 182.15234375 disk.rbd1.out‚ 13.4140625 disk.dm-0.in‚ 7567.421875# proc.memory.rssMB‚ 2772.74609375 cpu‚26.833929205712398 network.sent‚ 1.6371401e+07 disk.sda3.in‚0.109375 proc.cpu.threads‚17 network.recv‚ 1.715754e+06! disk.nvme0c0n1.in‚ 7567.421875( disk.nvme0c0n1.out‚13749.91943359375 disk.sda3.out‚ 182.15234375) proc.memory.percent‚16.92349910736084 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 182.15234375# memory_percent‚99.4617223739624 disk.nvme0n1.in‚ 7567.421875 disk.rbd0.out‚0& disk.nvme0n1.out‚13749.91943359375) disk./.usagePercent‚39.81963332225196 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0µ¬®p‘œ!‚(:† ÑÃíÒáÿØŠ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.603591918945312) gpu.0.memoryAllocatedBytes‚ 7390429184 gpu.0.temp‚31.0 gpu.0.powerWatts‚174.463' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.üû“á!‚(:Ö ÑÃíÒª³É% disk.nvme0n1.out‚13879.9912109375" disk.dm-0.out‚13879.9912109375 disk.sda.in‚0.109375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 182.37109375 disk.sda3.out‚ 182.37109375 disk.rbd0.in‚0 cpu‚26.83555604375769 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.109375 disk.drbd2165.out‚0) proc.memory.percent‚16.53416156768799 proc.cpu.threads‚17 network.sent‚ 1.6428513e+07 network.recv‚ 1.719956e+06$ disk./.usageGB‚593.3254661560059 disk.sda1.out‚0# proc.memory.rssMB‚ 2708.95703125 disk.nvme0n1.in‚ 7567.421875$ memory_percent‚98.80526065826416( proc.memory.availableMB‚ 195.74609375 disk.dm-1.in‚0.109375 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 7567.421875' disk.nvme0c0n1.out‚13879.9912109375* disk./.usagePercent‚39.818569866954725 disk.dm-0.in‚ 7567.421875 disk.sda.out‚ 182.37109375 disk.rbd1.out‚ 13.4140625ð˜úy2ž!- _wandbruntime‚5655  _runtime‚5655 ŒPÆŸ!À & lr-AdamW/pg1‚0.0006875057701355705 & lr-AdamW/pg2‚0.0006875057701355705 trainer/global_step‚12049 # _timestamp‚1.78437371399873e+09 _runtime‚5655.442581832 _step‚975MÃQË !Å trainer/global_step‚12049 # _timestamp‚1.78437371399873e+09 _runtime‚5655.442581832 _step‚975 & lr-AdamW/pg1‚0.0006875057701355705 & lr-AdamW/pg2‚0.0006875057701355705ÏÓVÆ°2¡!- _wandbruntime‚5655  _runtime‚5655 Öë£ë¢!ÓO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚5655.443813871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7116457537024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90534919 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468726973529600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90534917 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434416984825856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114432634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55602345 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90534919 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114442611 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7116545474048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27362435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90534917 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27362201 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434416959922176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461610515992576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7116457537024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114442611 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468727078505984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.smal%ê1îl_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204977530 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468726973529600 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚12049 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461610533031936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7351506374656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453855420618752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114432634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843737139998138e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚441768491200512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204967551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448sº•ºÞ'£!ØO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7091524230578 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82964783 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 300376064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460946841672880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55602348 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441768289992704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453855403579392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7091438093488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461610515992576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚327 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461610533031936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 306184192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 306184192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90534917 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27362201 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434416959922176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103176450 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461610515992576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7116457537024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114442611 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468727078505984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86137090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 300376064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204977530 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468726973529600 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6015668607 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6845104128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6538919936 trainer/global_step‚12049 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461610533031936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7351506374656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453855420618752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5739898880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114432634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6538919936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843737139998138e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87937024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚441768491200512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6845104128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204967551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 752447488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195816448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6538919936 _runtime‚5655.443813871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7116457537024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90534919 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚473 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468726973529600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90534917 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201207808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434416984825856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6845104128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 306184192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 298572143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚146 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6081670144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114432634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176304128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6081670144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87937024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5812080640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55602345 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚146 _step‚976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90534919 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114442611 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7116545474048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27362435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 779415040 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460983984206300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204994046 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90542140 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114451906 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114451906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114441915 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚12049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7352093498368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7092005415018 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453891892654080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453891875614720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204994046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468764758836736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461647626956800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460983881029738 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461647643996160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7117114840576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82971208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _runtime‚ 5655.7838795 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7117114840576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7117026896384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204984053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27364379 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461647626956800&7eŸå&¦!ßM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7352093498368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174199808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7092005415018 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6845104128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚473 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1346 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453891892654080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453891875614720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6538919936 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚473 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204994046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 752447488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468764758836736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461647626956800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460983881029738 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461647643996160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5812080640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7117114840576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82971208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195816448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103176562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86137202 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6538919936 _runtime‚ 5655.7838795 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 306184192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7117114840576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7117026896384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204984053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27364379 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚327 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461647626956800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87944192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 298572143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87944192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6015668607 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460983984206300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204994046 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90542140 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114451906 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6081670144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 300376064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114451906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114441915 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 779415040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚12049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104983552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3Ñ;¸ô%2§!- _wandbruntime‚5655  _runtime‚56555áäEÚ¨!ˆ 3loss/FBt2/assignment_loss‚0.0009416051395011603 /loss/FRt2/detection_loss‚1.4908411059635536 % _timestamp‚1.7843737143415046e+09 0loss/FRt2/assignment_loss‚0.7557757887543674 _runtime‚5655.7